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ABSTRACT 


Client  satisfaction  was  considered  to  be  a  critical  indicator  of  the  effectiveness  of 
the  services  provided  by  the  Navy  Regional  Data  Automation  Centers  (NARDACs). 
The  purpose  of  this  thesis  was  to  develop  the  means  to  measure  this  indicator. 

Interviews  of  twenty-eight  middle  management  clients  served  by  NARDACs  in 
four  geographical  regions  within  CONUS  were  conducted.  Forty-four  items  that 
influence  satisfaction  were  identified  belonging  to  eight  factor  dimensions  of  the 
client  satisfaction  domain.  The  Likert  Scale  methodology  was  employed  in  the 
construction  of  the  measurement  instrument.  The  instrument  was  reviewed  for 
completeness  of  items  and  logical  soundness  of  operations  by  three  independent 
groups  of  experts,  thus  achieving  content  validity.  A  client  satisfaction  index  is 
formulated  from  data  obtained  with  the  measurement  instrument. 

The  instrument  was  piloted  to  a  NARDAC  site  consisting  of  fifty-two  client 
organizations  using  self- administered  mail  surveys.  Evaluation  of  the  instrument 
was  performed  on  the  scored  results  of  twenty-six  pilot  respondents.  The  pilot 
results  were  subjected  to  empirical  tests  for  construct  validity  and  reliability.  The 
results  indicated  that  the  instrument  could  be  used  to  measure  client  satisfaction  after 
further  piloting  on  a  larger  (at  least  two  hundred  twenty)  sample  size. 
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I.     INTRODUCTION 


When  you  can  measure  what  you  are  speaking  about,  and  express  it  in 
numbers,  you  know  something  about  it;  but  when  you  cannot  express 
it  in  numbers,  your  knowledge  is  of  a  meager  and  unsatisfactory  kind; 
it  may  be  the  beginning  of  knowledge,  but  you  have  scarcely,  in  your 
thoughts,  advanced  to  the  stage  of  science.  [Cited  in  Ref .  1 :  p.  89] 


This  observation  by  the  renown  British  physicist,  William  Thompson  (Lord 
Kelvin),  may  have  been  foremost  in  the  minds  of  the  Naval  Data  Automation 
Command  (NAVDAC)  corporate  management  during  the  development  of  a  long-term 
business  strategy  for  their  field  activities,  the  Navy  Regional  Data  Automation  Center 
(NARDACs).  The  steering  committee,  composed  of  the  NARDAC  Commanding 
Officers  and  top  management  in  NAVDAC,  was  established  as  the  NARDAC  Board 
of  Directors  and  chartered  to  address  strategic  planning  matters  and  issues  affecting 
the  NARDACs  as  a  result  of  a  1986  Coopers  and  Ly brand  study  of  Navy  Industrial 
Fund  (NIF)  activities.^ 

One  of  the  issues  that  the  NARDAC  Board  of  Directors  identified  in  the 
corporate  long-range  business  plan  was  a  need  for  a  standardized  index  which  would 
be  used  to  measure  the  level  of  client  satisfaction  at  the  various  NARDACs.  The 
index  would  serve  the  purpose  of  providing  an  objective  evaluation  of  each 
NARDACs  performance  and  its  ability  to  deliver  quality  service  to  its  clients.  Also, 
as  a  measure  of  effectiveness,  the  index  would  be  the  cornerstone  of  their  business 


^  The  Secretary  of  the  Navy  (SECNAV)  contracted  Coopers  and  Ly  brand  to  conduct 
a  management  analysis  of  all  activities  under  the  NIF  program.  The  purpose  of  the  study 
was  to  assess  the  NIF  program's  effectiveness  and  to  find  any  areas  of  deficiency. 


strategy  to  increase  the  NARDAC's  competitive  posture.  This  would  be 
accomplished  by  accurately  evaluating  their  current  level  of  client  satisfaction  to 
enable  the  establishment  of  a  base  index  from  which  management  can  take 
appropriate  actions  to  make  improvements. 

Presently,  a  standardized  measurement  of  effectiveness  that  empirically 
represents  user  satisfaction  among  those  clients  serviced  by  the  NARDACs  does  not 
exist.  Each  NARDAC,  operating  as  a  cost  center,  is  responsible  for  marketing  its 
services  to  activities  normally  located  within  its  geographical  area  of  cognizance.  As 
a  result,  the  NARDACs  currently  assess  client  satisfaction  using  different  methods, 
criteria,  standards  and  metrics.  This,  in  effect,  has  made  it  difficult  for  NAVDAC  to 
obtain  an  accurate  assessment  of  client  satisfaction  at  each  of  their  NARDACs. 

A .  EVOLUTION  OF  THE  REQUIREMENT 
1.     Background 

The  NARDACs  were  formed  as  a  result  of  the  Navy  Automated  Data 
Processing  (ADP)  Reorganization  study  and  implementation  plan  with  the  basic 
objective  to  improve  the  management  and  operation  of  the  Navy's  mission  support 
ADP  program  [Ref.  2].  These  field  activities  of  NAVDAC  were  established  to 
provide  non-tactical  ADP  technical  assistance  and  operational  support  to  Navy 
activities  ashore  and  afloat.  Department  of  Defense  (DOD)  components  and  other 
Federal  agencies.  There  are  nine  (9)  NARDACs  geographically  dispersed  within  the 
continental  United  States  (CONUS)  and  in  Hawaii  (See  Figure  1.1).  Although  each 
NARDAC  may  vary  in  the  ADP  support  it  can  provide,  major  functional  services  are 
offered  at  every  site  (See  Appendix  A). 
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Figure  1.1  NARDAC  Installations  Map 


2.  History  of  Operations 

Prior  to  fiscal  year  1984,  the  NARDACs  were  mission-funded  which 
meant  that  every  NARDAC  received  an  annual  allotment  of  appropriated  funds  to 
cover  its  costs  for  the  support  provided  to  its  clients.  Based  on  that  operational 
philosophy,  ADP  products  and  services  were  delivered  at  no  cost  to  the  clients. 
Because  this  ADP  support  was  perceived  by  the  clients  as  free,  there  was  little 
concern  on  their  part  for  costs,  only  the  timeliness  and  quality  of  the  requested 
support.  In  that  regard,  the  NARDACs'  primary  mission  was  focused  on  providing 
ADP  support  to  their  clients  within  the  planned  budgets.  The  need  to  formally 
measure  client  satisfaction,  therefore,  was  not  a  paramount  issue  at  the  time. 

3.  Present-Day  Operations 

Since  fiscal  year  1984,  the  NARDACs'  cost  of  operations  has  been 
financed  under  the  Navy  Industrial  Fund  (NIF),  a  revolving  fund  from  which 
working  capital  funded  by  Congress  to  the  Department  of  Defense  is  allocated  to  the 
Navy  as  an  appropriation.  The  initial  funds  by  the  NIF  appropriation  to  the 
NARDACs  were  used  to  finance  the  costs  of  providing  ADP  services  ordered  by 
their  clients.  In  return,  the  products  furnished  and  services  rendered  by  the 
NARDACs  to  other  Navy  activities  and  government  agencies  are  performed  on  a  cost 
reimbursable  basis  at  standardized  fiscal  year  rates.^  It  is  therefore  the  responsibility 
of  the  client  to  submit  a  Program  Objective  Memorandum  (POM)  and  budget 


1  Standardized  fiscal  rates  refer  to  stabilized  rates  for  products  and  services  that  are 
fixed  for  an  entire  fiscal  year.  The  period  of  a  fiscal  year  is  from  1  October  to  30 
September. 


accordingly  for  needed  ADP  services. ^  The  clients  pay  the  NARDACs  through  the 
use  of  a  reimbursable  order  citing  their  own  appropriated  funds.  In  theory,  the 
payment  or  reimbursement  would  then  put  the  corpus  of  the  revolving  fund  back  to 
where  it  started.  The  financial  operations  of  the  NARDACs  are  comparable  to  that  of 
private  enterprises,  i.e.  working  capital  and  clients  are  required.  The  NARDACs  and 
other  NIF  activities,  however,  are  run  on  a  non-profit  basis. 

Working  with  a  NIF  activity  is  much  like  contracting  with  a  contractor 
~  the  only  significant  difference  is  that  the  NEF  activity  is  not  out  to 
make  a  profit  for  stockholders.  [Ref.  3:  p.  H22] 

Moreover,  the  importance  of  how  effective  a  NARDAC  operates  cannot  be 

understated.   The  less  effective  the  NARDAC,  the  higher  the  charge  for  services 

[Ref.  3:  p.  HI]. 

According  to  CDR  Charles  Taylor,  Commanding  Officer  of  NARDAC 

Pearl  Harbor,  the  change  to  NIF  was  made  for  two  major  reasons.  First,  in  partial 

emulation  of  enterprises  in  the  commercial  sector,  the  intention  was  to  discipline  and 

motivate  the  NARDACs  to  provide  the  best  possible  service  or  risk  going  out  of 

business.  The  second  major  reason  was  to  provide  a  measure  of  discipline  to  the 

users.  If  services  remained /re^  to  the  users,  as  was  the  case  when  the  NARDACs 

were  under  mission  funding,  there  would  be  no  incentive  for  the  users  to  request 

only  those  services  critical  for  mission  accomplishment.  Together,  these  two  factors 

were  intended  to  guarantee  more  efficient  use  of  ADP  resources  in  the  Navy. 


1  The  POM  is  a  DOD  Planning,  Programming  and  Budgeting  System  (PPBS) 
document,  prepared  by  the  Services  to  the  Secretary  of  Defense,  which  expresses  total 
program  requirements  in  terms  of  force  structure,  manpower,  material  and  costs  to  satisfy 
assigned  functions  and  responsibilities  during  the  period  of  the  Five  Year  Defense 
Program.  [Ref3:p.  All] 


4.     Impact  of  NIF  on  Operations 

It  was  clearly  obvious  to  the  NARDACs  that  the  transition  to  NIF  forced  a 
significant  change  in  their  way  of  doing  business.  NIF  had  introduced  new  factors 
into  the  NARDACs'  system  of  operations  which  had,  until  then,  only  applied  to 
comparable  organizations  in  the  commercial  sector.  The  principal  change  in  direction 
was  now  focused  on  the  clientele  from  whom  revenues  to  continue  operations  were 
obtained.  In  addition  to  providing  quality  ADP  support,  the  issue  of  operating  cost- 
effectively  and  to  remain  reasonably  affordable  for  the  clients,  became  of  paramount 
importance.  Moreover,  unlike  other  NIF  activities  who  maintain  a  quasi-monopoly 
over  the  products  and  services  they  provide,  the  NARDACs  were  now  forced  to 
compete  with  other  government  agencies,  as  well  as  the  private  sector,  to  provide 
non-tactical  ADP  support. 

Because  of  the  changes  driven  by  NIF,  client  satisfaction  has  become 
critical  for  the  NARDACs.  Since  all  services  are  provided  on  a  cost  reimbursable 
basis  and  all  operating  expenses  are  paid  for  by  revenues  generated  by  the  clients,  the 
NARDACs  must  maintain  a  positive  image  and  provide  quality  service  in  order  to 
retain  their  share  of  the  market.  A  client  satisfaction  index,  therefore,  could  serve  as 
a  tool  by  which  the  NARDACs  can  gauge  their  market  leverage  and  consequently 
make  whatever  adjustments  necessary  to  remain  competitive  and  financially 
operational. 

B.  OBJECTIVES 

The  objectives  of  this  thesis  are  to  identify  the  attributes  relating  to  client 
satisfaction  and  to  develop  a  standardized  methodology  for  measuring  and  analyzing 
these  attributes  through  the  construction  of  a  prototype  measurement  instrument  and 
client  satisfaction  index  for  subsequent  piloting  and  review  at  the  NARDACs. 


C.  FOCAL  RESEARCH  QUESTIONS 

The  focal  issues  of  this  thesis  are: 

What  is  an  acceptable  definition  of  a  satisfied  client? 

What  attributes  should  be  used  to  construct  a  client  satisfaction  index  for  a 
NARDAC? 

How  should  the  data  be  gathered  and  in  what  form? 

What  metrics  should  be  used  to  measure  and  analyze  the  data  and  how  can 
these  metrics  be  validated? 

Can  a  microcomputer  be  used  for  data  analysis  and  for  maintaining 
information  on  customer  satisfaction?  If  so,  what  software  should  h>e  used? 

How  can  the  data  gathering  and  analysis  process  be  institutionalized  to  make  it 
part  of  an  ongoing  effort  to  meet  client  needs? 

D.  SCOPE,  LIMITATIONS  AND  ASSUMPTIONS 

The  scope  of  this  thesis  is  limited  to  the  development  of  a  standardized  prototype 
client  satisfaction  index  specifically  tailored  to  represent  an  empirical  measurement  of 
client  satisfaction  for  the  nine  NARDACs.  The  NARDACs  are  located  in  CONUS  at 
Naval  Air  Station  (NAS)  Alameda  -  for  NARDAC  San  Francisco,  NAS  North  Island 
-  for  NARDAC  San  Diego,  New  Orieans,  NAS  Pensacola,  NAS  Jacksonville,  NAS 
Norfolk,  Washington  Navy  Yard  -  for  NARDAC  Washington,  and  Newport;  and  in 
Hawaii  at  Naval  Station  Pearl  Harbor.  Each  NARDAC  has  a  unique  set  of  clients 
within  its  geographical  region.  The  research  focuses  primarily  on  those  attributes 
relating  to  client  satisfaction  as  ascertained  from  a  cross-section  of  NARDAC  clients. 

E.  ORGANIZATION  OF  THE  THESIS 

Chapter  II  presents  a  literature  review  pertinent  to  current  thinking  regarding 
client  satisfaction,  in  private  industry  and  in  government. 


Chapter  III  describes  the  methodology  employed  in  the  development  of  a 
measurement  instrument  designed  as  the  basis  for  the  formulation  of  the  client 
satisfaction  index. 

Chapter  IV  presents  an  evaluation  of  the  validity  and  reliability  of  the  client 
satisfaction  questionnaire  as  a  measurement  instrument. 

Chapter  V  describes  the  construction  of  the  Client  Satisfaction  Index  (CSI)  and 
several  other  methods  of  analyzing  the  data  collected  from  the  questionnaire. 

Chapter  VI  presents  the  conclusions  of  the  study  and  recommendations  for 
implementation  and  further  research  .  This  chapter  concludes  the  thesis. 


II.    LITERATURE  REVIEW 

A.  INTRODUCTION 

The  theme  of  this  year's  American  Marketing  Association's  annual  meeting  was 
"The  Customer  Is  King."  The  meeting  focused  on,  and  stressed  the  importance  of 
attracting  and  retaining  customers.  Many  corporations  and  management  consultants 
have  come  to  the  same  realization  that  the  key  to  maximizing  profits  is  maximizing 
customer  satisfaction  (e.g.,  [Ref.  4:  pp.  2-5],  [Ref  5:  pp.  13-16],  [Ref.  6:  p.  5]). 
During  the  past  five  years  of  financial  deregulation,  banking  strategists  have 
increasingly  concluded  that  one  of  a  bank's  most  important  assets  is  its  existing 
customer  base  [Ref.  7:  pp.  6,  57].  IBM  believes  that  satisfied  customers  are  the 
basis  of  their  continued  business  success  [Ref.  8].  In  the  words  of  Darryl 
Landvater,  President  of  Oliver  Wright  Video  Production,  Inc., 

Although  there  are  other  ways  to  get  an  additional  five  percent  of  sales 
to  the  bottom  line,  many  of  them  such  as  cutting  expenditures  for 
labor,  equipment,  and  research  and  development  are  both  painful  and 
potentially  damaging.  It's  easier  to  boost  sales  and  earnings  through 
superior  customer  service  and  at  the  same  time  improve  the  the  long- 
term  position  of  the  company.  [Ref.  9:  p.  86] 

Paul  Allaire,  President  of  Xerox,  has  recently  made  customer  satisfaction  it's 

"first  priority",  with  return  on  assets  (ROA)  and  increased  market  share  second  and 

third  respectfully  [Ref.  4:  p.  2].  In  a  memo  to  his  division  directors,  Allaire  says, 

"we  can  only  achieve  our  ROA  and  market  share  goals  by  satisfying  our  customers." 

Xerox's  new  focus  on  customer  satisfaction  was  the  basis  for  the  cover  story, 

"Customer  Satisfaction:  The  Big  Payoff  in  Xerox's  quarterly  corporate  publication 

Benchmark  [Ref.  4].  In  this  article,  five  leading  management  consultants  expound 


on  the  importance  of  satisfying  the  customer  and  its  payoff  in  increased  profits.  In 
the  article,  Robert  Watemian,  Jr.,  co-author  of  In  Search  of  Excellence,  cited  a  study 
from  the  Strategic  Planning  Institute  in  Cambridge,  Massachusetts. 

Business  sectors  with  a  higher  service  content  have  a  higher  return  on 
investment.  The  institute,  which  ranked  similar  companies  in  order  by 
the  relative  quality  of  their  products  and  services,  found  that  the 
highest  rated  company  in  each  area  had  a  return  on  investment  100 
percent  higher  than  the  lowest-ranked  company.  [Ref.  4:  p.  3] 

Christopher  Lovelock,  a  former  Harvard  professor  and  principal  of  Christopher 
Lovelock  and  Associates,  states  that  the  key  to  improving  customer  satisfaction  "...is 
with  research  to  measure  what  characteristics  of  services  or  products  are  important  to 
users  as  well  as  how  customers  perceive  the  goals  provided  by  the  company."  [Ref. 
4:  p.  3] 

Tom  Peters,  co-author  of  In  Search  of  Excellence  and  A  Passion  for  Excellence, 
beUeves  that  the  qualitative  aspects  of  business  must  be  quantified.  He  believes  in  the 
old  adage  "What  gets  measured  gets  done."  This  philosophy  is  applied  rigorously 
within  his  organization  and  at  his  executive  seminars  in  the  following  manner. 


In  the  customer  arena,  we  believe  that  regular  quantitative  measurement 
of  customer  satisfaction  provides  a  much  better  lead  indicator  of  future 
organizational  health  than  does  profitability  or  market-share  change. 
We  suggest  monthly  measurement.  Further,  we  urge  participants  to 
make  the  level  of  customer  satisfaction  the  primary  basis  for  incentive 
compensation  and  annual  performance  evaluation  for  virtually  every 
person  at  every  level  in  every  function  throughout  the  organization. 
[Ref.  10:  p.  El] 

Clearly,  customer  satisfaction  is  an  issue  which  is  receiving  considerable 

attention  in  the  corporate  community  and  recently  it  has  been  gaining  momentum  in 

the  computer  services  industry.  The  purpose  of  this  literature  review  is  to  survey 

some  of  the  recent  research  that  has  been  conducted  in  the  area  of  customer 
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satisfaction  with  computer  services  and  the  instruments  which  were  developed  to 
quantify  and  measure  satisfaction.  The  literature  review  is  divided  into  three 
sections.  The  first  section,  Measures  of  Computer  User  Satisfaction,  is  a  review  of 
generic  measures  which  could  be  used  at  any  computer  services  organization  or 
management  information  system/data  processing  (MIS/DP)  department.  The  second 
section.  Survey  of  Current  Industry  Measures,  is  a  review  of  current  measures  of 
customer  satisfaction  used  in  the  computer  industry.  And  the  last  section. 
Department  of  Defense  Measures,  is  a  brief  look  at  what  kinds  of  measures  of 
customer  satisfaction  are  being  used  within  the  Department  of  Defense. 

B .  MEASURES  OF  COMPUTER  USER  SATISFACTION 

The  largest  initial  obstacle  in  conducting  this  literature  review  was  finding 
information  related  to  the  measurement  of  customer  satisfaction  at  a  computer 
services  organization.  The  problem  was  the  word  "customer".  The  computer 
industry  and  MIS/DP  departments,  in  many  instances,  still  refers  to  its  customers  as 
users  instead  of  valuable  customers  [Ref.  11:  p.  286].  According  to  W.H.  Inmon, 
"Universally,  data  processing  exists  as  a  service  organization  within  the  company. 
The  service  is  for  the  user,  and  the  satisfaction  of  the  user  ultimately  determines  the 
success  or  failure  of  data  processing."  [Ref.  12:  p.  224] 

There  have  been  several  studies  conducted  recently  on  the  concept  of  computer 
user  satisfaction  or  user  information  satisfaction  (UIS).  In  the  words  of  livari,  UIS 
refers  to  "...a  cluster  of  concepts  that  imply  the  assessment  of  information  systems  or 
information  services  in  the  user's  subjective  terms."  [Ref.  13:  p.  57]  The 
foundations  of  UIS  are  based  on  the  research  by  Cyert  and  March  [Ref.  14].  The 
essential  concept  is  "...that  an  information  system  which  meets  the  needs  of  its  user 
will  reinforce  satisfaction  with  that  system.    If  the  system  does  not  provide  the 
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needed  informarion,  the  user  will  become  dissatisfied  and  look  elsewhere  ."  [Ref.  15: 
p.786] 

One  of  the  first  measures  of  UIS  related  to  the  total  MIS/DP  function  in  an 
organization  which  has  received  considerable  attention  was  developed  by  S.  W. 
Pearson  [Ref.  16].  During  his  research,  Pearson  compiled  a  list  of  thirty-six  factors 
relating  to  computer  user  satisfaction  based  on  a  literature  review  of  twenty-two 
studies  of  computer-user  interactions.  These  factors  were  then  reviewed  by  three 
DP  professionals  for  completeness  and  accuracy.  As  a  result  of  the  review,  two 
additional  factors  were  added  to  the  list.  Next,  the  list  of  thirty-eight  factors  was 
compared,  using  a  critical  incident  analysis  technique,  to  taped  interview  responses 
from  thirty-two  middle  manager  users  in  eight  different  organizations.  This  step 
concluded  with  the  addition  of  one  factor,  making  a  total  of  thirty-nine  distinct 
factors.  These  factors  were  then  incorporated  into  a  questionnaire  which  utilized  the 
semantic  differential  technique.  Each  factor  is  measured  by  three  separate  scales 
using  seven  intervals  from  negative  to  positive  responses.  The  first  scale  was  a 
measure  of  one's  perception  by  rating  four  bipolar  adjective  pairs.  The  second  scale 
was  a  satisfactory  -  unsatisfactory  pair  and  the  third  scale  measured  the  importance  of 
the  factor  to  the  user.  The  importance  rating  is  used  as  a  weighting  factor  for  the 
overall  satisfaction  score.  The  description  of  the  scoring  method  can  be  found  in 
Reference  17.  The  questionnaire  was  empirically  tested  for  validity  and  reliability 
using  the  same  thirty-two  middle  managers  previously  interviewed.  The  instrument 
was  found  to  be  reliable  and  valid,  based  on  the  twenty-nine  returned  questionnaires. 
See  Reference  17  for  the  results  of  the  validity  and  reliability  evaluation  of  the 
questionnaire. 
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A  study  by  Ives,  Olson  and  Baroudi  was  a  continuation  of  the  Pearson  study 
with  the  emphasis  to  reduce  the  length  of  the  overall  measure  and  reinforce  the 
validity  of  the  instrument  with  more  extensive  testing  [Ref.  15].  The  results  of  their 
study  reduced  Pearson's  thirty-nine  original  factors  down  to  twenty-one  factors  with 
greater  validity  using  a  larger  sample  size.  Description  of  their  results  can  be  found 
in  [Ref.  15].  Ives,  Olson,  and  Baroudi's  work  was  based  on  the  following 
definition  of  UIS:  "...the  extent  to  which  users  believe  the  information  system 
available  to  them  meets  their  information  requirements." 

Mathew  &  Co.,  a  Data  Processing  Management  consulting  firm,  has  developed  a 
reporting  system  for  measuring  DP-user  satisfaction  called  "How're  We  Doing?" 
[Ref.  18:  p.  10]  The  system  requires  users  to  annotate  in  a  log,  during  a  one  month 
period,  any  problems  with  data  processing  based  on  nine  "performance-related 
criteria."  At  the  end  of  the  month,  the  data  from  the  log(s)  are  processed  by  Mathew 
&  Co.  which  supplies  the  client  with  DP  user  satisfaction  analyses  in  several 
different  categories.  The  system  is  based  on  exception  reporting  (i.e.,  only  problems 
with  data  processing  are  reported).  The  nine  criteria  are  timeliness  of  output,  quality 
of  output,  online  availability,  response  time,  systems  development  schedules, 
response  to  problems,  and  attitude  and  cooperativeness.  Mathew  &  Co.  provides  its 
clients  with  reports  on  DP-user  trends  for  the  current  month  and  trends  over  periods 
of  three  months,  six  months  and  twelve  months. 

C.  SURVEY  OF  CURRENT  INDUSTRY  MEASURES 

The  banking  industry  recently  began  using  a  customer  service  index  to  measure 
and  manage  customer  service  [Ref.  7].  One  index  is  based  on  twenty  critical 
attributes  of  "Good  Customer  Service"  according  to  a  nationwide  survey  of  bank 
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customers.  Bank  customers  were  asked  to  score  all  twenty  attributes  of  their 
financial  institution  on  a  six-  point  scale,  (6  =  excellent). ^ 

Keeping  the  customer  satisfied  with  quality  products  and  services  is  the 
challenge  changing  the  face  of  the  automotive  industry  also.  The  automotive  industry 
uses  a  consumer  satisfaction  index  to  evaluate  consumer  demands  (e.g.,  increasing 
showroom  hours).   [Ref.  19] 

Sitmar  Cruises  attributes  its  highest  rate  of  repeat  passengers  of  any  major  cruise 
line  to  its  "intensive  customer  research"  using  a  16-page  customer  survey  [Ref.  4:  p. 
4].  Benjamin  French  believes  random  surveys  are  the  best  way  of  determining  the 
effectiveness  of  an  organization's  customer  satisfaction  program. 

A  brief  questionnaire  and  accompanying  letter  of  explanation  are 
simple  to  prepare  and  provide  an  inexpensive  way  of  taking  the  pulse 
of  your  customers.  In  addition  to  providing  you  with  data  on  your 
effectiveness,  the  survey  also  conveys  your  concern  for  solving 
customer's  problems  and  gives  you  another  opportunity  to  restore  their 
faith  in  your  company  and  its  products  or  services.  [Ref.  20:  p.  Ill] 

However,  James  Carman,  Professor  of  Business  Administration  at  the  Graduate 
School  of  Business,  University  of  California  at  Berkeley,  argues  that  surveys  must 
be  interpreted  carefully  because  it  is  difficult  for  some  customers  to  evaluate  the 
quality  of  the  service  they  received  if  they  are  lacking  knowledge  in  the  field.  [Ref. 
4:  p.  5] 

IBM  distributes  a  survey  to  every  one  of  their  customers  at  least  once  a  year  in 
order  to  directly  ascertain  information  relating  to  customer  satisfaction  and  to  improve 
their  business  relationship.  The  corporate  offices  send  out  periodic  surveys  to  clients 
requesting  their  service  perceptions.  This  is  done  independendy  from  the  regional 


^  The  index  was  developed  by  Financial  Products  Group,  a  Chicago-based  consulting 
firm  serving  the  financial  services  industry,  with  the  assistance  of  Market  Facts,    ic. 
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offices.  In  fact,  regional  offices  don't  even  know  when  this  occurs.  The  survey  is 
divided  into  eight  major  categories:  quality  of  products,  hardware  maintenance  and 
support,  systems  management,  marketing  support,  technical  support,  education, 
telephone  coverage,  and  overall  rating.  The  survey  is  scored  on  a  five-point  Likert 
Scale  (very  satisfied  to  very  dissatisfied)  with  a  block  for  no  opinion.  In  addition  to 
the  survey,  there  is  a  corporate-wide  complaint  system  where  client  complaints  are 
assigned  to  a  high-level  manager  for  investigation  and  follow-up.  [Ref.  8] 

At  Electronic  Data  Systems  Corporation  (EDS),  they  believe  feedback  from  the 
customer  is  an  important  step  in  maintaining  a  high  level  of  customer  satisfaction. 
EDS  North  American  Commercial  Group  uses  an  extensive  series  of  questionnaires 
as  part  of  their  Quality  Enhancement  Program  (QEP)i  to  keep  abreast  of  their 
customers  needs  and  to  continue  to  improve  services  and  products.  [Ref.  21] 

D.  DEPARTMENT  OF  DEFENSE  MEASURES 

The  Philadelphia  Naval  Shipyard,  a  NIF-funded  activity,  does  not  use  any 
proactive  initiatives  to  ascertain  customer  satisfaction.  Their  criteria  for  measuring 
performance  is  based  on  adhering  to  project  budget  and  schedule  constraints^.  The 
Military  Airlift  Command  (MAC),  an  Air  Force  industrial  funded  activity,  does  not 
actively  solicit  information  from  their  customers  regarding  customer  satisfaction 
either.  Instead,  suggestion/comment  forms  are  made  available  to  MAC  customers.^ 


^  The  contents  of  the  questionnaire  and  the  specifics  of  the  QEP  implementation 
policies  and  procedures  are  proprietary  information  of  EDS  North  American  Commercial 
Group  and  will  not  be  disclosed  in  this  thesis. 

2  Interview  with  Mr.  Monteleon,  Management  Planning  Division,  Philadelphia  Naval 
Shipyard,  Philadelphia,  Pennsylvania,  5  February  1988. 

3  Interview  with  CDR  Jordan,  USN,  Navy  Liaison  Officer,  Military  Airlift  Command 
Headquarters,  Scott  Air  Force  Base,  5  February  1988. 
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The  Navy  Public  Works  Centers  (PWC),  which  are  NIF-funded  activities,  have 
recently  initiated  a  customer  satisfaction  evaluation  program  [Ref.  22].  The  program, 
"Customer  Evaluation  of  Support  Provided  by  PWC",  consists  of  a  fifty-item 
questionnaire  divided  into  six  categories.  The  respondent  is  requested  to  rate  the 
quality  of  service  provided  by  the  PWC  using  a  seven-point  scale.  The  questionnaire 
is  scored  in  the  following  manner: 

a.  For  customer  responses  where  the  "X"  is  placed  in  blocks  6  or  7, 
score  2  points; 

b.  For  customer  responses  where  the  "X"  is  placed  in  blocks  3,  4,  or 
5,  score  1  point; 

c.  For  customer  responses  where  the  "X"  is  placed  in  blocks  1  or  2, 
score  0  points. 

The  best  possible  score  is  100  points.  To  determine  the  annual  grade 
from  all  of  the  customers  who  received  a  questionnaire,  add  the  total 
questionnaire  scores  and  divide  by  the  total  number  of  questionnaires 
that  were  completed  in  the  year.  The  total  score  is  weighted  by 
customer  size  (i.e.  four  surveys  from  large  customers,  two  from 
medium-  sized  customers  and  one  each  from  all  others.)  The  result  is  a 
summarized  customer  evaluation  of  the  support  provided  by  the  PWC 
in  the  past  year. 

We  found  no  published  studies  performed  by  the  Department  of  the  Navy  or  the 

Department  of  Defense  in  the  area  of  customer  or  user  satisfaction  relating  to 

computer  services  organizations  or  MIS/DP  departments.^ 


^  Searches  were  made  through  the  Defense  Logistics  Studies  Information  Exchange 
(DLSIE),  Defense  Technical  Information  Center  (DTIC),  and  DIALOG  Information 
Services,  Inc. 


16 


III.  QUESTIONNAIRE  DEVELOPMENT 

The  corporate  heads  of  the  NARDACs  identified  the  critical  requirement  for  a 
client  satisfaction  index,  thus  setting  the  stage  for  the  next  step  of  our  research.  We 
needed  to  identify  a  logical  process  to  follow  in  order  to  arrive  at  an  index  that 
accurately  represents  the  client's  satisfaction  with  the  NARDACs  performance  in 
providing  products  and  services.  To  represent  that  state  of  satisfaction  as 
unequivocally  as  possible,  it  was  imperative  that  the  theme  of  our  methodology  be 
based  on  the  inputs  from  actual  NARDAC  clients  and  their  composite  view  of  client 
satisfaction.  This  was  important  because  it  ensured  that  the  fundamental  building 
blocks  for  the  construction  of  a  meaningful  index  were  obtained  directly  from 
representative  samples  of  the  source.  By  using  this  approach,  we  felt  that  our 
methodology  would  be  sound  in  terms  of  face  validity  and  the  resulting  product 
would  be  realistic  and  more  readily  accepted  by  the  NARDACs  and  their  client 
community. 

From  a  study  by  Pearson,  the  measurement  of  client  satisfaction  mirrors  the 
measurement  of  an  individual's  attitude  or  evaluative  feelings  toward  some  feature  of 
the  subjects  of  interest  [Ref.  16].  There  are  various  ways  to  collect  data  for 
measuring  a  client's  attitude.  One  basic  means  is  by  interrogation  which  involves 
getting  people  to  answer  questions,  either  in  person  or  through  telephone  interviews. 
Another  interrogation  technique,  more  widely  used  for  measuring  a  person's  attitude, 
involves  an  instrument,  in  most  cases  a  questionnaire,  which  is  one  of  the 
fundamental  tools  used  in  survey  research.  The  reason  for  the  popularity  of  the 
survey  research  method  is  due  to  the  numerous  benefits  it  offers. 
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•  It  provides  a  method  for  empirical  verification  of  data. 

•  The  data  gathered  by  this  method  becomes  a  source  of  information  which  can 
be  conveniently  stored  through  automated  methods  and  can  be  analyzed  shortly 
after  collection. 

•  The  responses  can  be  coded  into  a  standardized  form  for  recording  in  a 
quantitative  manner;  therefore  a  standardized  measurement  that  is  consistent 
across  respondents  is  achieved  and  lends  itself  to  the  development  of  an  index, 

•  Surveys  can  be  administered  fairly  readily  by  an  implementing  organization  (a 
NARDAC)  and  can  be  flexibly  designed  with  minimum  impact  on  the 
respondents'  (the  clients')  time.  ([Ref.  23]  and  [Ref.  24]) 

It  is  because  of  these  advantages  and  its  applicability  to  the  measurement  of  client 

satisfaction  that  the  survey  research  approach  was  selected  for  this  study. 

A.  BACKGROUND 

During  the  preliminary  stages  of  the  research,  background  information  on  the 
NARDACs  was  gathered  in  order  to  obtain  an  understanding  of  their  organizational 
structure  and  hierarchy,  corporate  culture  and  environment,  products  and  services, 
and  client  information.  This  background  information  was  obtained  during  site  visits 
at  NARDAC  San  Francisco,  NARDAC  Washington  and  NAVDAC.  Further  data 
was  received  from  NARDAC  Pearl  Harbor  and  NARDAC  San  Diego.  Additionally, 
the  marketing  representative  at  Honeywell  was  interviewed  to  see  how  client 
satisfaction  was  viewed  from  the  industry  perspective.  We  also  reviewed  literature 
on  customer  satisfaction  and  found  current  trends  in  the  field,  particularly  in  service- 
oriented  industries. 

At  the  same  time,  we  were  given  information  on  existing  customer  satisfaction 
methods  employed  by  the  NARDACs  and  Public  Works  Centers  (PWC).  Primarily, 
the  methods  used  by  these  communities  are  informal  and  based  on  feedback  from  the 
client,  usually  obtained  during  personal  visits  or  phone  conversation.  The  PWCs 
and  several  NARDACs  employed  more  formal  methods,  using  questionnaires  that 
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were  developed  in-house,  as  part  of  their  marketing  efforts.  Some  of  these  methods, 
extracted  from  a  summary  of  findings  [Ref.  25],  are  listed  below. 

•  NARDAC  Jacksonville  developed  a  client  assessment  form  which  is  used 
during  quarterly  client  visits.  This  assessment  seems  to  be  the  most  formal 
means  employed  to  gather  client  satisfaction  information  from  the  client's 
perspective. 

•  NARDAC  San  Diego  identified  specific  client  satisfaction  indices  with  it's 
mainframe  clients.  These  indices  are:  "95%  products  on  time"  and  "98%  on- 
line availability  during  prime  time  and  off-shift  hours."  Prime  time  availability 
is  particularly  significant  to  its  major  client,  the  Naval  Aviation  Depot.  These 
indices  are  assessed  and  reported  during  regular  client  meetings. 

•  NARDAC  San  Francisco  recently  developed  an  information  form  which  will 
be  used  by  client  relations  personnel  when  visiting  clients.  A  portion  of  this 
form  provides  clients  the  opportunity  to  express  satisfaction  with  or  concerns 
about  services. 

•  NARDAC  Pensacola  and  NARDAC  San  Diego  use  written  reports  of  visits  or 
phone  contacts  to  identify  client  problems  or  concerns.  These  reports  also 
provide  vehicles  to  convey  the  cUent's  satisfaction  with  services. 

•  NARDAC  Washington  developed  a  quarterly  rating  report  which  is  filled  out 
by  NARDAC  Washington  department  directors.  This  report  provides  the 
department  directors'  expectation  of  a  client's  response  to  various  service 
satisfaction  questions. 


Because  of  the  wide  diversity  in  methods  used  among  the  NARDACs,  it  is 
understandable  why  the  NARDAC  Board  of  Directors  recognized  the  need  for  a 
standardized  method  of  obtaining  critical  client  satisfaction  information. 

B .  FACTOR  IDENTIFICATION 

Using  the  information  gathered  from  interviews  with  NARDAC  management 
personnel,  relevant  organizational  documents  and  literature  searches,  we  generated  a 
list  of  the  major  products  and  services  that  the  NARDAC  provides  to  its  customers. 
From  this  list,  an  initial  set  of  factors,  which  was  seen  as  measuring  the  NARDACs' 
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performance  in  delivering  products  and  services,  was  compiled.    This  initial  set 
consisted  of  the  following  nine  (9)  factors: 

Computer  capabilities 

Ease  of  obtaining  services 

System  design  and  programming  expertise 

Technical  support 

ADP  consultation  and  assistance 

Training 

Cost  of  ADP  services 

Customer  support 

Timeliness  and  quality  of  products  and  services  provided 

In  contrast,  the  factors  used  in  existing  NARDAC  questionnaires  [Ref.  25]  included: 

Response  to  client  requests  (the  quality  of  response  and  the  rapidity  of 
response) 

Thoroughness  of  information  conveyed  to  clients 

NARDAC  service  accessibility  to  clients 

Cost 

Perception  of  the  rehabiUty  of  NARDAC  personnel 

Non-recurrence  of  problems  and  willingness  to  refer  other  organizations  to 
NARDAC  for  computing  services 

C .  CONTENT  VALIDATION 

In  order  to  validate  the  accuracy  of  our  initial  set  of  factors  and  thereby  establish 
content  validity,  a  method  was  needed  which  was  expedient,  allowed  for  personal 
interaction  with  existing  clients  to  verify  the  assumptions  made  in  compiling  the  list 
of  factors,  and  permitted  the  opportunity  to  identify  other  aspects  of  the  NARDAC- 
client  relationship  which  may  have  been  overlooked  or  omitted  during  previous 
interviews  with  NARDAC  personnel  and  documentation  review.  It  was  decided  then 
that  interviews  with  a  representative  group  of  active  NARDAC  clients  would 


20 


accomplish  this  objective.    A  series  of  interviews  with  clients  served  by  the 
NARDACs  would  be  conducted  within  research  Umitations. 
1.     Interview  Procedures 

Because  of  time  constraints  and  a  ceiling  on  travel  expenses,  it  was 
necessary  to  set  criteria  for  determining  our  interview  sample.  This  criteria  included 
location,  types  of  client  organizations  to  interview,  who  within  the  client  organization 
to  interview  and  number  of  interviews  to  conduct. 

The  first  and  foremost  criterion  was  location.  We  wanted  to  focus  on 
regions  having  a  large  concentration  of  Navy  activities  with  differing  ADP  support 
requirements  and  where  major  NARDAC  installations  supporting  these  Navy 
commands  offered  a  wide  variety  of  computer  services.  The  East  Coast  and  West 
Coast  regions  appeared  to  satisfy  this  criterion  and  offered  the  best  representative 
cross-section  of  respondents.  The  clients  to  be  selected  for  interview  were  those 
supported  by  NARDAC  San  Francisco  and  NARDAC  San  Diego  on  the  West  Coast 
and  NARDAC  Washington  and  NARDAC  Norfolk  on  the  East  Coast. 

The  next  criterion  was  types  of  chents  to  consider  for  the  interviews.  Since 
the  objective  in  this  phase  of  the  research  was  to  obtain  information  on  what  factors 
affect  or  influence  client  satisfaction,  we  wanted  to  interview  current/active  clients 
who  maintained  some  frequency  of  interaction  with  the  NARDAC  providing  them 
services.  One  way  of  determining  the  relative  degree  of  interaction  was  to  examine 
the  amount  of  revenue  generated  from  the  client  during  the  last  fiscal  year.  This  data 
was  readily  available  to  the  Client  Liaison  Officer  at  each  NARDAC  who  compiled  a 
ranked  listing  of  clients  (by  revenues  earned)  from  which  candidate  clients  were 
selected.  Although  interviews  with  previous  clients  who  had  elected  not  to  renew 
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services  with  the  NARDAC  were  considered,  we  were  unable  to  include  that  client 
category  under  this  criterion  because  of  time  constraints. 

Another  criterion  used  in  narrowing  the  sample  size  of  clients  for  the 
interview  was  selection  of  the  person  in  the  client  organization  with  whom  to  conduct 
the  interview.  We  determined  that  the  best  candidate  to  satisfy  this  criterion  was  the 
individual  designated  by  the  client  organization  as  the  command's  official  liaison  or 
point  of  contact  (POC)  with  the  NARDAC.  This  individual  would  usually  belong  in 
middle  management,  serving  as  the  activity's  ADP  Officer.  Since  POC  information 
normally  resided  alongside  the  client  organization  data,  this  information  was 
maintained  and  kept  current  by  the  NARDAC's  Chent  Liaison  Officer. 

The  last  criterion,  number  of  interviews  to  conduct,  was  constrained  by  a 
number  of  factors: 

•  Number  of  days  available  to  conduct  the  interviews 

•  Availability  of  the  clients  for  the  interview 

•  Length  of  the  interview 

•  Travel  time  window  required  between  interviews  (during  normal  working 
hours) 

In  most  cases,  the  Client  Liaison  Officer  of  the  NARDAC  was  in  a  better  position  to 
juggle  these  constraints  and  therefore  coordinated  appointments  with  the  clients  and 
arranged  the  interview  schedule.  This  schedule  included  interviews  with  28  client 
organizations,  the  list  of  which  is  summarized  in  Appendix  B.  Each  of  us 
interviewed  the  client  organization's  POC,  some  of  whom  brought  in  members  of 
their  staff  to  provide  input  during  the  session. 

After  the  appointments  with  the  clients  were  scheduled,  the  Client  Liaison 
Officer  provided  us  a  list  containing  the  names  and  addresses  of  the  clients  to  be 
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interviewed.  Shortly  thereafter,  we  contacted  each  client  by  phone  to  personally 
introduce  ourselves  and  to  explain  the  nature  of  our  research.  Following  these  phone 
calls,  we  mailed  each  client  a  letter  containing  the  purpose  of  the  interview,  in 
particular,  to  obtain  their  views  regarding  the  factors  that  they,  as  clients,  would 
use  to  evaluate  the  services  provided  by  their  NARDAC.  A  sample  letter  can  be 
found  in  Appendix  C.  A  week  before  the  interviews,  we  again  called  each  client  to 
verify  receipt  of  the  letter  and  to  confirm  the  date  and  time  of  our  interview.  This  call 
also  gave  the  client  the  opportunity  to  ask  specific  questions  about  the  interview  or 
our  research  and  allowed  for  any  last  minute  rescheduling  of  the  appointment  if 
required. 

To  ensure  efficient  use  of  time  during  the  interview  and  to  facilitate  data 
collection/recording,  forms  were  used  which  we  had  prepared  beforehand  to  guide  us 
through  the  interview.  These  forms  served  to  assist  us  in  collecting  background 
information  regarding  the  clients  (see  Appendix  D)  and  their  views  on  what  particular 
factors  contributed  to  their  satisfaction  with  NARDAC  services  (see  Appendix  E).  In 
addition,  the  forms  were  used  to  ensure  consistency  in  the  types  of  information  asked 
of  the  client.  In  certain  instances,  interviews  were  taped  but  not  before  receiving 
permission  from  the  client.  In  no  instance  was  permission  denied. 
2.     Factor  Validation 

During  the  interviews,  we  asked  each  client  to  comment  on  the  nine  factors 
which  we  had  previously  compiled  during  the  process  described  in  Section  B  and  to 
indicate  whether  each  factor  influenced  their  satisfaction  with  the  NARDAC's 
performance  in  providing  services.  The  client  was  also  asked  to  identify  any  other 
factors  which  were  not  included  in  the  original  set  and  did  influence  their  state  of 
satisfaction.    We  then  requested  each  client  to  rank  all  the  factors  by  order  of 
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importance.  In  some  cases,  clients  were  strongly  inclined  to  group  several  factors 
equally  important  under  the  same  ranking.  Other  clients  felt  that  some  of  the  factors 
were  not  pertinent  to  their  current  requirements  and  therefore  excluded  those  factors 
from  their  ranking. 

Following  the  completion  of  all  the  interviews,  we  compiled  a  tabulated  list 
of  factors  that  were  ranked  important  to  client  satisfaction  by  each  client.  The 
compiled  list  provided  us  an  across  the  board  ranking  of  all  the  factors  to  help  in 
determining  which  factors  were  considered  most  important  by  the  clients.  To  obtain 
a  composite  ranking  of  the  factors,  we  averaged  the  rankings  given  by  the  clients  on 
each  factor.  The  composite  ranking  of  the  top  six  factors  included: 

Ranking  Factor 


1 


Timeliness  and  quality  of  products  and  services 

2  Technical  support 

3  System  design  and  programming  expertise 

4  Ease  of  obtaining  services 

5  Customer  support 

6  Cost  of  ADP  services 


We  ascertained  that  the  highest  ranked  factor  was,  in  fact,  two  separate 
factor  dimensions.  Timeliness  and  Quality.  By  incorporating  these  dimensions  as 
separate  factors,  a  modified  list  of  seven  factors  was  obtained.  The  remaining  factors 
in  the  original  set  were  not  discounted.  They  were  determined  to  be  too  specific  in 
scope  and  were  therefore  considered  to  be  candidate  subelements  (items)  relating  to 
one  of  the  seven  factors. 
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3.     Item  Development  and  Validation 

Our  next  step  was  to  identify  specific  items  obtained  from  the  interviews 
which  related  to  the  NARDAC's  performance  and  which  were  measurable.  The 
information  was  extracted  from  client  comments  documented  on  the  form  we  used 
during  the  interviews  (see  Part  4  of  Appendix  E)  and  obtained  in  taped  sessions. 
Working  with  our  own  individual  set  of  interview  data,  we  placed  each  item  on  a  3 
by  5  card.  When  this  was  completed,  we  compared  our  sets  of  items  for  redundancy 
and  clarification,  while  referring  to  information  from  other  surveys  and  relevant 
literature.  Our  sets  were  then  consolidated  into  a  set  containing  forty-two 
performance-related  items  which  were  then  categorized  under  one  of  the  seven 
factors  we  had  identified  in  the  previous  process.  In  those  instances  when  items 
could  not  be  placed  under  a  category  because  of  the  factor's  specificity,  we  redefined 
the  scope  of  the  factor  and  broadened  its  applicability  to  include  the  item.  The 
factors,  as  modified  by  this  process,  are  listed  below. 

Timeliness  of  Products  and  Services 

Quality  of  Products  and  Services 

Technical  Proficiency 

Hardware,  Software  and  Communications  Technology 

Accessibility 

Customer  Support 

Service  Level  Management 

The  final  set  of  items  and  modified  factors  established  the  client  satisfaction 

domain  for  the  NARDACs  and  provided  the  fundamental  basis  of  what  to  measure  in 

order  to  gauge  the  level  of  an  individual  client's  satisfaction.  The  key  elements  of 

this  set,  which  is  presented  in  Appendix  F  and  discussed  in  the  following  sections, 

formed  the  foundation  for  the  construction  of  the  measurement  instrument. 
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D.  MEASUREMENT  INSTRUMENT 

Several  different  measuring  instruments  were  investigated  through  literature 
searches  and  reviewing  surveys  currently  being  used  in  the  government  and  private 
industry  to  determine  which  instrument  would  best  suit  our  study.  Buzzell,  Cox  and 
Brown  assert,  the  more  structured  the  measuring  instrument  is,  the  more  accurate  the 
output  [Ref.  26].  Typically,  in  marketing  measurements,  it  is  important  to  have  a 
measure  of  the  degree  of  the  response.  To  aid  in  this  type  of  measurement,  degrees 
of  response  are  scaled  to  make  more  or  less  discriminations.  Three  of  the  most 
common  marketing  measurements  are  variants  of  the  Semantic  differential  scale, 
Likert  scales  and  Paired  comparisons  (see  Figure  3.1).  According  to  Buzzell,  Cox 
and  Brown,  however. 

Although  it  is  possible  to  attach  numbers  to  such  a  verbal  scale  for 
purposes  of  analysis,  it  should  be  remembered  that  this  is  still  an 
ordinal  scale,  strictly  speaking,  and  estimates  with  regard  to  intervals 
or  degrees  of  response  are  simply  estimates.  Quantify  them  if  you 
will,  but  there  is  no  direct  means  of  determining  the  validity  of  such 
quantifications. 

There  are  several  methods  of  collecting  data  for  the  instrument.  The  most 
common  ones  used  in  marketing  research  are  personal  interviews,  telephone 
interviews  and  self-administered  questionnaires  (mail  surveys).  Listed  in  Figure  3.2 
below  are  some  of  the  advantages  and  disadvantages  of  each  method,  as  cited  by 
([Ref.  27]  and  [Ref.  28]).  The  surveys  that  we  found  currently  being  used  in  the 
government  and  industry  were  all  self-administered  questionnaires.  In  the 
government,  cost  and  time  restraints  seem  to  make  self-administered  mail 
questionnaires  more  prevalent. 

1.     Questionnaire  Development  Standards 

Due  to  the  limited  resources  available  at  a  NARDAC  to  conduct  surveys, 
self-administered  questionnaires  would  be  the  easiest  to  implement.  There  are 
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SEMANTIC  DIFFERENTIAL  SCALE 

"Would  you  rate  General  Motors  as  being: 

Progressive 

Strong 


1 


1 


Conservative 


Weak?" 


LIKERT  SCALE 

"Ajax  is  an  excellent  cleanser." 


Agree  extremely  strongly 

Agree  fairly  strongly 

Agree 

Undecided 

Disagree 

Disagree  fairly  strongly 

Disagree  extremely  strongly 


PAIRED  COMPARISONS,  e.g.. 

Do  you  prefer  Brand  A  or  Brand  B? 
Do  you  prefer  Brand  C  or  Brand  A? 
Do  you  prefer  Brand  B  or  Brand  C? 


Figure  3.1  Commonly  Used  Types  of  Rating  Methods  and  Scales. 
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1.  Personal   interviews  : 

Advantages: 

•  They  tend  to  allow  the  use  of  a  more  representative  sample. 

•  Achieve  higher  return  rate. 

•  Produce  fewer  incomplete  questionnaires. 

•  More  questions  can  be  asked. 

•  More  complex  measurement  methods  can  be  used. 

•  Verification  or  responses  may  t>e  made  more  readily. 

Disadvantages: 

•  More  costiy. 

•  Subject  to  interviewer  bias,  error  and  cheating. 

•  Subject  to  response  bias. 

2.  Telephone   interviews  : 

Advantages: 

•  Can  be  conducted  quickly. 

•  Relatively  low  cost. 


Disadvantages: 

•  Sample  bias. 

•  Usually  must  be  brief. 

3.    Self -Administered   questionnaires: 

Advantages: 

•  Least  costly. 

•  Avoid  interviewer  bias. 

•  Larger  number  of  respondents  can  be  reached. 

•  More  convenient  to  the  respondent. 

•  Requires  a  smaller  staff  for  administering. 

Disadvantages: 

•  Sample  is  almost  certainly  not  likely  to  be  "representative"  unless  follow 
up  is  done. 

•  Must  be  carefully  designed  and  pretested  to  avoid  confusion  on  the  part 
of  the  respondent. 


Figure  3.2     Data  Collection  Methods  (Advantages  and  Disadvantages) 
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several  methods  of  conducting  a  self-administered  questionnaire.  The  questionnaire 
could  be  administered  to  a  group  of  respondents  gathered  at  the  same  place  at  the 
same  time  (e.g.,  at  a  NARDAC  Client  Council/Advisory  Board  meeting).  They 
could  also  be  hand-delivered  at  the  respondent's  office  to  be  completed  and  picked  up 
at  a  later  time.  Mailing  is  another  option,  or  a  combination  of  mailing  and  one  of  the 
above  methods  can  also  be  used.  [Ref.  23:  p.  159]  The  questionnaire  can  be  hand- 
delivered,  where  it  could  be  explained  and  any  questions  answered,  and  then  mailed 
when  completed.  The  questionnaire  designed  for  the  NARDACs  was  based  on  the 
premise  that  the  survey  would  be  mailed  with  limited  intervention  required  by  the 
NARDAC.  The  following  criteria  was  used  in  developing  the  NARDAC  Client 
Satisfaction  questionnaire: 

Instructions  for  completing  the  questionnaire  must  be  clear  and  concise. 

Scales  must  be  easily  understandable  and  unambiguous. 

Questions  will  be  closed-ended  and  as  easy  to  answer  as  possible. 

Definitions  will  be  provided  as  required. 

Comment  section  will  be  provided. 

The  number  of  questions  should  be  held  to  an  absolute  minimum. 

Questionnaire  must  have  an  overall  professional  appearance  (i.e.,  neat  and 
legible). 

Since  the  majority  of  the  questionnaire  respondents  are  considered  middle 

to  upper  level  management  within  their  organization  (OS- 12  to  GM-15),  we  felt  a 

self-administered  questionnaire  would  provide  us  with  the  needed  information  with 

minimum  intrusion  on  their  schedules.  In  this  regard,  we  felt  it  was  important  that 

the  instructions  were  self-explanatory  and  the  questions  easily  answered.  According 

to  Arlene  Fink,  "A  self-administered  questionnaire  that  is  hard  to  read  can  confuse  or 

irritate  respondents.   The  result  is  a  loss  of  data."  [Ref.  28:  p.  44]    We  tried  to 
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formulate  the  questions  in  the  respondents'  own  vernacular.  Definitions  of  certain 
temiinology  and  phrases  were  provided  to  help  alleviate  any  ambiguities.  We  also 
felt  it  was  important  to  provide  the  respondent  with  an  opportunity  to  comment  on 
any  of  the  items  in  the  questionnaire  or  provide  additional  information  since  the 
questions  were  closed-ended.  Additionally,  the  questionnaire  had  to  make  a  good 
visual  impression.  Since  the  questionnaires  are  distributed  to  valued  NARDAC 
clients,  we  felt  that  if  the  appearance  made  a  bad  first  impression,  then  response  rate 
could  possibly  suffer. 

2.  Questionnaire  Format 
The  questionnaire  measures  the  respondents  level  of  satisfaction  with  respect  to  a  set 
of  forty-two  performance  related  items.  These  items  are  grouped  together  into  seven 
categories  or  factors  as  described  in  section  B  (Factor  Identification)  above. 
Although  we  were  concerned  with  the  length  of  the  questionnaire,  we  allowed 
adequate  space  between  questions  for  comments  and  to  prevent  the  questionnaire 
from  looking  cluttered.  The  questionnaire  is  eight  pages  long  or  four  pages  copied 
back-to-back.  It  is  widely  acknowledged  in  the  social  research  community  that  an 
improperly  laid  out  questionnaire  can  not  only  confuse  the  respondents,  but  also 
make  the  scoring  of  the  questionnaire  more  error-prone  and  time  consuming.  One  of 
the  most  common  questionnaire  formats  is  one  where  the  respondent  is  asked  to 
check  one  response  from  a  series  [Ref.  27:  p.  205].  Each  item  is  scored  using  a  five- 
point  Likert  scale.  A  not  applicable  box  is  also  provided.  According  to  Babbie, 
boxes,  adequately  spaced  apart,  are  the  best  for  the  respondent  to  answer.  The  scale 
is  measured  from  one  to  five,  where  five  is  very  satisfied,  and  one  is  very 
dissatisfied.  The  respondents  are  asked  to  place  an  X  in  the  box  which  most 
appropriately  describes  their  level  of  satisfaction  with  that  item.  The  scales  are  laid 


30 


out  vertically  to  the  side  of  each  item  to  facilitate  scoring  of  the  questionnaire.  Key 
words  and/or  phrases  were  underlined  in  each  item  to  help  focus  the  respondent's 
attention  on  the  major  point  the  item  is  measuring  (see  Appendix  F). 
3.     Questionnaire  Critique 

After  the  first  draft  of  the  questionnaire  was  completed,  it  was  sent  out  for 
review  to  all  nine  NARDAC  Client  Liaison  Officers/Marketing  representatives,  and  to 
various  Naval  Postgraduate  School  faculty  (see  Appendix  C).  Generally,  the 
comments  from  the  NARDACs  were  favorable.  Most  felt  we  had  developed  a 
comprehensive  questionnaire  and  had  addressed  the  major  issues  presently 
concerning  the  NARDAC  clients.  The  majority  of  the  comments  focused  on  the 
format  and  length  of  the  questionnaire.  Although  some  of  the  NARDACs  indicated 
there  were  some  questions  that  might  be  redundant,  none  of  them  indicated  which 
ones  were  the  culprits.  One  NARDAC  felt  the  sentence  structure  of  the  items  was 
too  choppy  and  short,  and  another  suggested  we  include  an  overall  rating  question  at 
the  end  of  the  questionnaire.  Surprisingly,  the  majority  of  the  comments  regarding 
item  construction  came  from  the  Naval  Postgraduate  School  faculty.  It  was  pointed 
out  that  many  of  our  items  were  asking  the  respondent  to  rate  more  than  one  aspect  in 
a  single  item.  For  example,  in  item  twelve,  the  respondent  is  asked  to  rate  \h&  format 
and  the  quality  of  the  Chargeback  Report.  The  respondent  may  feel  that  the  format  is 
satisfactory,  but  the  quality  is  not.  Other  comments  dealt  with  the  lack  of  definitions 
regarding  the  meaning  of  certain  terms,  and  the  order  in  which  the  questions  were 
presented.  The  above  comments  were  incorporated  into  the  second  draft  of  the 
questionnaire.  Comments  referring  to  the  implementation  of  the  questionnaire  were 
disregarded  because  they  fell  outside  the  scope  of  our  study  (e.g.,  adding  an  internal 
routing  stamp). 
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4.     Questionnaire  Second  Draft  (Short  Form) 

The  second  draft  of  the  questionnaire  looked  considerably  different  from 
the  first  version  (see  Appendix  G).  It  is  divided  into  three  major  sections.  The  first 
section  still  measures  the  respondents  level  of  satisfaction  with  respect  to  a  set  of 
performance  related  items,  but  the  three  comment  lines  have  been  dropped,  two 
additional  questions  were  added,  and  the  ordering  and  grouping  of  the  questions 
were  changed.  These  revisions  made  the  questionnaire  appear  to  be  shorter  -  four 
pages  long  or  two  pages  copied  back-to-back  (Short  Form  ).  The  items  are  still 
grouped  together  into  seven  categories  or  factors,  but  the  names  have  changed. 
Although  the  length  of  the  questionnaire  was  a  concern,  we  did  allow  adequate 
spacing  between  questions  to  prevent  the  questionnaire  from  looking  cluttered.  The 
response  boxes  formed  a  matrix  down  the  right  side  of  the  page  to  facilitate  the 
scoring  of  the  questionnaires.  The  scale  was  repeated  at  the  top  of  each  page  for  the 
convenience  of  the  respondent. 

A  new  section  was  also  added  to  the  Short  Form.  In  Robert  AUoway's 
paper.  Defining  Success  for  Data  Processing,  he  discovered  that  in  the  six  companies 
that  he  studied,  all  were  violating  the  fundamental  rule  of  management,  "...identify 
which  activities  are  most  important  and  allocate  resources  to  ensure  good 
performance  on  those  activities."  [Ref.  29:  p.  1]  Thus,  the  second  section  of  the 
questionnaire  measures  the  relative  importance  of  each  of  the  seven  major  categories 
described  in  the  first  section  of  the  questionnaire.  Again,  each  category  is  scored 
using  a  five-point  interval  scale.  Boxes  are  provided  for  the  respondents  answers. 
The  scale  is  measured  from  one  to  five,  where  five  is  critical,  and  one  is  irrelevant. 
The  use  of  the  second  section  will  be  described  in  detail  in  Chapter  V.  The  data  from 
this  section  will  not  be  used  in  the  computation  of  the  index;  it  will  only  be  used  as  a 


32 


management  tool.   The  last  section  of  the  questionnaire  permits  the  respondent  to 
make  comments  about  any  item(s)  or  general  comments  about  the  service  provided 
by  the  NARDAC.  The  trailing  edge  of  the  questionnaire  provides  the  NARDAC  with 
pertinent  administrative  information  regarding  the  respondent. 
5.     Final  Review  of  Questionnaire 

A  revised  long  form  ^  and  the  short  form  were  sent  out  to  all  NARDAC 
Commanding  Officers  (COs)  and  NAVDAC  for  one  last  review  before  the  pilot 
survey  was  conducted.  During  a  NARDAC  Board  of  Directors  meeting,  the  COs 
decided  that  the  scale  identifiers  at  each  extreme  did  not  demonstrate  enough 
differentiation  from  the  other  identifiers.  Therefore,  the  words  extremely  satisfied 
and  extremely  dissatisfied  replaced  the  identifiers  very  satisfied  and  very 
dissatisfied.  They  also  changed  the  middle  identifier  from  Neither  Satisfied  Nor 
Dissatisfied  to  Neutral.  They  felt  this  wording  would  be  be  easier  to  interpret  by  the 
respondents  and  NARDAC  management.  Other  changes  that  were  made  included  the 
definition  of  factors  and  items,  the  rearranging  and  regrouping  of  items,  and  the 
revision  of  some  of  the  factor  names.  One  additional  factor  was  also  added.  The 
short  form  of  the  questionnaire  was  unanimously  approved.  They  felt  that  the  long 
form  would  be  too  overwhelming  for  most  of  their  clients.  However,  they  still 
would  like  to  use  the  long  form  for  specific  clients  from  whom  more  data  is  required 
to  help  identify  and  resolve  discrepancies. 


1    The  revised  long  form  is  the  original  questionnaire  with  onlv  content  changes 
made.  The  format  remained  the  same  (see  Appendix  H). 
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IV.     QUESTIONNAIRE  EVALUATION 

This  chapter  describes  the  methods  we  employed  to  evaluate  the  measurement 
instrument,  the  client  satisfaction  questionnaire.  As  discussed  in  the  preceding 
chapters,  this  instrument  was  developed  to  capture  the  domain  of  client  satisfaction 
for  the  NARDACs  and  the  distinguishable  elements  (items)  that  influence  this 
domain.  The  administration  of  the  questionnaire  to  a  group  of  subject  clients  at  a 
pilot  site  and  the  evaluation  of  the  questionnaire  based  on  the  scored  results  are 
presented.  The  results  were  analyzed  using  several  statistical  software  packages. 
STATGRAPHICS,  a  personal  computer  (PC)  based  statistical  graphics  system,  and 
the  Statistical  Package  for  Social  Scientists  (SPSS-X),  a  mainframe  computer 
program,  were  used  in  performing  descriptive  statistics,  construct  validation  tests, 
and  reliability  tests.  In  addition,  various  microcomputer  programs  were  augmented 
to  provide  other  tools  for  the  evaluation  of  the  measurement  instrument. 

A.  PILOT  DESCRIPTION 

To  collect  the  data  for  evaluation  of  the  questionnaire,  we  used  a  self- 
administered  mail  survey.  One  NARDAC  site  was  chosen  to  pilot  the  measurement 
instrument,  consisting  of  fifty-two  client  organizations  which  formed  the  sample 
population.  Although  we  realize  that  a  nationwide  survey  would  provide  us  with  a 
greater  sample  size  to  item  ratio  and  more  statistically  sound  test  results,  time  and 
schedule  constraints  and  the  lack  of  physical  resources  compelled  us  to  preclude 
exercising  this  alternative.  Nevertheless,  we  felt  the  pilot  sample  size  was  sufficient 
for  an  initial  evaluation  of  the  instrument.  Considering  geographical  proximity,  the 
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clients  serviced  by  NARDAC  San  Francisco  were  therefore  selected  as  the  subject 
group  for  evaluation  of  the  questionnaire  (see  Appendix  I). 

1.  Pilot  Administration 

After  receiving  approval  from  the  Commanding  Officer  of  NARDAC  San 
Francisco  to  conduct  the  pilot,  we  obtained  a  list  of  active  clients  from  his  Client 
Liaison  staff.  Information  from  this  list  included  the  name  of  the  client  organization, 
name  of  the  organization's  POC,  mailing  address  and  telephone  number. 
Additionally,  the  staff  supplied  us  with  supplemental  profile  information  on  each 
client.  The  profile  data  consisted  of  current  fiscal  year  revenues  earned  to  date  from 
the  clients  and  types  of  services  provided  to  the  clients  by  the  NARDAC. 

Once  the  NARDAC  Board  of  Directors  gave  us  the  approval  to  proceed 
with  the  pilot  testing,  we  mailed  each  client  a  pilot  survey  package  that  consisted  of  a 
cover  letter,  the  self-administered  questionnaire  (see  Appendix  J)  and  a  pre- 
addressed  return  envelope.  In  the  cover  letter,  we  requested  each  client's  cooperation 
in  completing  the  questionnaire  to  assist  us  in  evaluating  the  measurement 
instrument.  The  client  was  also  asked  to  critique  the  questionnaire  in  terms  of 
content,  presentation  and  format.  A  sample  of  the  cover  letter  is  presented  in 
Appendix  C. 

The  survey  packages  were  posted  to  the  clients  ten  days  before  the 
requested  return  deadline  which  allowed  the  respondents  at  least  one  full  work  week 
to  complete  and  return  the  questionnaire.  The  return  envelope  was  provided  to 
accelerate  the  return  time. 

2.  Pilot  Response 

Of  the  fifty-two  survey  packages  mailed,  we  received  twenty-six  completed 
questionnaires  which  seemed  to  suggest  a  fifty  percent  response  rate.  Current  survey 
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research  guidelines  on  percentage  return  rates  indicate  that  fifty  percent  is  an 
acceptable  response  rate.  Babbie  suggests  that  a  response  rate  of  seventy  j)ercent  or 
more  is  very  good,  a  response  rate  of  at  least  sixty  percent  is  good  and  a  response 
rate  of  at  least  fifty  percent  is  adequate  for  analysis  and  reporting.  He  does  caution, 
however,  that  this  yardstick  is  only  a  rough  guide  which  has  no  statistical  basis.  The 
accepted  practice  in  computing  response  rates  is  to  omit  all  those  questionnaires  that 
could  not  be  delivered  due  to  bad  addresses  and  the  like.  This  number  is  subtracted 
from  the  initial  sample  size  to  obtain  the  net  sample  size.  Then,  the  number  of 
completed  questionnaires  is  divided  by  the  net  sample  size  to  produce  the  response 
rate.  [Ref.  23:  p.  165]  Although  the  number  of  questionnaires  we  received  was  far 
below  the  number  we  expected,  we  felt  that  this  could  have  been  attributed  to  the 
timing  of  the  pilot  survey  which  occurred  during  the  Christmas  season  (early 
December).  To  substantiate  reasons  for  adjusting  the  net  sample  size,  it  was 
necessary  for  us  to  obtain  concrete  evidence  for  the  number  of  omitted  non- 
responses.  To  accomplish  this,  we  made  numerous  attempts  to  contact  by  phone 
those  clients  who  had  not  responded.  In  many  cases,  we  were  unsuccessful  in  our 
efforts  due  to  busy  signals  or  no  answer.  In  those  few  instances  when  we  were  able 
to  contact  the  client  organization,  messages  were  left  to  the  organization's  POC,  yet 
no  return  calls  were  ever  received.  Some  of  the  client  organization  POCs  who  were 
successfully  contacted  indicated  that  they  never  received  the  survey  package.  In  one 
specific  case,  the  survey  package  had  been  addressed  to  the  client  organization 
headquarters  rather  than  to  the  field  activity  where  the  organization's  POC  was 
located.  Among  the  non-respondents  contacted,  we  accounted  for  three  clients  who 
did  not  receive  the  survey  package.  Two  survey  packages  were  returned  for  reasons 
of  insufficient  address  or  unknown  addressee.     And  lastly,  one  completed 
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questionnaire  arrived  four  weeks  after  our  analysis  of  the  initial  set  of  twenty-six 
responses.  Our  net  sample  size  after  adjustment,  therefore,  was  forty-six  which 
yielded  a  response  rate  of  fifty-seven  percent. 

Due  to  time  and  schedule  constraints,  subsequent  mailings  were  not 
conducted.  Moreover,  we  determined  that  the  response  rate  of  fifty-seven  percent 
was  acceptable  for  purposes  of  our  evaluation. 
3.     Processing  of  Pilot  Data 

The  twenty-six  returned  questionnaires  were  processed  in  the  following 
manner.  As  survey  packages  were  returned  and  after  each  envelope  was  opened,  the 
questionnaire  was  reviewed  and  checked  against  the  master  mailing  list.  Each 
completed  questionnaire  was  then  assigned  a  unique  client  code.  The  convention 
used  in  assigning  the  client  code  was  straightforward  -  a  number  that  represented  the 
sequence  in  which  the  questionnaire  was  received.  The  scored  results  of  each 
questionnaire  were  then  entered  into  a  microcomputer  based  spreadsheet  (EXCEL) 
and  saved  on  diskettes  for  subsequent  processing  by  various  statistical  software 
programs.  The  specific  software  programs  and  the  results  obtained  from  the 
processing  of  the  scores  are  discussed  in  the  next  following  sections. 

B.  DESCRIPTIVE  STATISTICS 

Basic  descriptive  statistics  were  used  to  evaluate  the  data  collected  from  the  pilot 
survey.^  Frequency  distributions  were  performed  on  each  item  in  order  to  uncover 
any  irregularities  in  the  responses.  What  was  found,  was  an  unusually  large  number 
of  not  applicable  (N/A)  responses  had  been  given  for  many  of  the  items.  Figure  4.1 


1    The  descriptive  statistics  developed  from  the  pilot  survey  are  proprietary 
information  of  NARDAC  San  Francisco,  and  will  not  be  disclosed  in  this  thesis. 
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shows  a  histogram  of  all  the  items  and  their  associated  percent  of  not  applicable 
responses.  A  three-step  process  was  used  to  determine  the  cause  of  the  high  N/A 
response  rate. 

The  first  step  was  to  isolate  those  items  that  had  an  unusually  large  number  of 
not  applicable  responses;  the  mean  was  chosen  as  the  break-point.  Any  item  which 
had  a  not  applicable  response  rate  above  31.4  percent  was  coded  with  an  asterisk  on 
the  histogram  as  shown  in  Figure  4.1  (i.e.,  items  5,  6,  9,  17,  18,  19,  20,  21,  22, 
23,  25,  26,  27,  29,  38,  and  42).  Table  4.1  shows  the  break-down  of  the  isolated 
items  by  factor. 

It  was  interesting  to  note  that  all  of  the  items  for  the  factor  Quality  of  Products 
and  Services,  and  three  out  of  the  five  items  for  the  factor  Timeliness  of  Service,  fell 
above  the  mean  (3 1 .4%)  N/A  response  rate.  Problems  discovered  in  item 
construction  for  these  two  factors  will  be  discussed  in  Section  D  of  this  chapter. 
The  next  step  was  to  see  if  there  were  any  relationships  between  the  sixteen  items 
listed  in  Table  4.1  and  the  respondents  which  contributed  significantly  to  the  N/A 
response  rate  for  those  items.  Figure  4.2  shows  the  number  of  N/A  responses  for 
the  sixteen  isolated  items  by  respondent.  Again,  the  mean  was  chosen  as  the  break- 
point to  separate  those  respondents  that  had  a  significantly  large  number  of  N/A 
responses.   The  asterisks  in  Figure  4.2  indicate  which  respondents  fell  above  the 
mean  (9)  rate  (i.e.,  respondents  1,  2,  3,  4,  5,  7,  8,  11,  12,  13,  21,  22,  23,  24,  and 
25). 

During  the  last  step,  we  looked  at  the  profile  of  the  respondents  isolated  in  the 
previous  step  to  determine  what  may  have  caused  these  respondents  to  answer  N/A  to 
so  many  items.  Table  4.2  lists  all  the  pilot  respondents  (by  code),  NARDAC  San 
Francisco's  fiscal- year-to-date  (FYTD)  revenue  for  that  respondent,  and  the  type  of 


39 


EL 


!fic\i 


CNi 


£ 


•o 


oil 

6 

■■^ 

a> 


2        CO 


LU 
O 


ei 


u 


u 


o 

a. 

CO 

LU 

cc 


21 


91 


U 


I    •    I 


CNJ 


O  0>  <D  (O 


CM 


SIAI31I  a3i\nosi 


c 

03 
0) 


> 

O 

CO 

< 


c 
•c 
o 
u 
CO 

(A 

c 

(U 

■o 

c 

8. 

en 

cr 


o 


CQ 

(A 

c 

o 
a 

en 
a> 

< 
Z 

b< 

3 
01) 


40 


TABLE  4.2  Pilot 

Client  Profile  Data 

CLIENT  CODE 

FYTD  REVENUE  ($000) 

SERVICE  PROVIDED 

/ 

23 

Risk    Assessment 

2 

5 

Training 

3 

2 

Study 

4 

1 

Programming 

5 

0 

• 

6 

46 

Training  &  Procurement 

7 

2 

Training 

8 

17 

Training 

9 

10 

IRC  Support/Sperry 

10 

5 

Labor 

11 

14 

Study  &  Procurement 

12 

1 

Programming 

13 

117 

A  DP  Security  Risk 
Assesment 

14 

26 

Programming 

15 

24 

IRC  Support,  Procurement,  Tech 
Support 

16 

172 

Programming,  Telecom,  Key- 
Entry,  Burroughs,  IV  Phase 

17 

6 

FYPR-Sperry 

18 

34 

Sperry,  Comlines,  Terminal 

19 

1 

S  perry 

20 

5 

Labor,  Misc,  Telecom 

21 

21 

Procurement,    Training 

22 

8 

IRC  Support,  Procurement 

23 

293 

Programming,   Procurement 

24 

6 

Training 

25 

0 

- 

26 

17 

IRC  Support,  Procurement 

Mean 

32.9 

For  all  respondents 

Mean 

7.7 

(Except  for  respondents  13 
&  23) 

Sum 

856 

For  all  respondents 
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service  provided.  The  respondents  in  bold  type  are  the  ones  isolated  in  step  two. 
The  following  observations  were  made: 

•  The  mean  FYTD  Revenue  for  all  the  isolated  respondents  (except  respondents 
13  and  23y  was  $7,700.  As  compared  to  the  total  mean  for  all  respondents, 
these  are  relatively  small  clients. 

•  Forty  percent  of  the  highlighted  respondents  only  received  training  as  a 
service.  The  questionnaire  does  not  accommodate  training  services  as  well  as 
other  services  provided  by  the  NARDAC.  This  problematic  area  will  be 
addressed  in  Section  D  of  this  chapter.  It  should  also  be  noted  that  many  of  the 
NARDACs  distribute  separate  surveys  for  clients  receiving  training. 

•  The  contract  for  respondent  3  had  not  begun  at  the  time  the  pilot  was 
conducted. 

•  The  contracts  for  respondents  5  and  25  have  been  cancelled. 

TABLE  4.1       Break-down  of  Isolated  Factors. 


FACTOR 

ITEM(S) 

Accessibility 

5 

System  Resources 

6,9 

Cost  Management 

- 

Quality  of  Products  and  Services 

17,  18,  19,20,21,22,23 

Timeliness  of  Services 

25,  26,  27 

Responsiveness 

29 

Staffing 

38 

General  Business  Practices 

43 

^  Respondent  13  had  a  one  time  contract  for  ADP  Security  Risk  Assessment  and  had 
little  interaction  with  NARDAC  San  Francisco.  Respondent  23  had  a  large  hardware 
procurement  which  accounted  for  75%  of  his  total  FYTD  revenue.  It  was  felt  that  these 
two  respondents'  relatively  high  FYTD  revenue  was  not  the  norm  and  were  disregarded  in 
the  computation  of  the  mean  for  the  isolated  respondents. 
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It  would  be  premature  to  make  any  widespread  conclusions  about  the  quality  of 
the  questionnaire  from  this  phase  of  the  evaluation  since  the  number  of  respondents 
(26)  from  the  pilot  survey  was  small.  Further  testing  of  the  instrument  with  a  larger 
sample  size  will  be  required  in  order  properly  evaluate  the  questionnaire;  however,  it 
is  clear  that  there  are  problems  with  some  of  the  items  in  the  factors  Quality  of 
Products  and  Services  and  Timeliness  of  Services.  These  issues  will  be  addressed  in 
Section  D. 

Generally,  the  comments  about  the  questionnaire  were  quite  favorable.  Only 
seven  out  of  the  twenty-six  respondents  actually  commented  on  the  construction  of 
the  questionnaire.  Of  those  who  commented,  over  seventy  percent  thought  the 
questionnaire  was  well  prepared.  A  typical  comment  was,  "Your  questionnaire  was 
quite  thorough,  easy  to  read,  and  concise.. .the  instructions  were  fine...".  Only  one 
respondent  (#12)  felt  the  questionnaire  was  too  long,  but  he  is  a  new  client  and 
answered  thirty-one  of  the  forty-four  items  on  the  questionnaire  with  a  not  applicable 
response. 

C.  VALIDATION  AND  RELIABILITY  PROCEDURES 

The  next  step  in  the  evaluation  of  the  instrument  was  to  test  the  questionnaire  for 
its  validity  and  reliability.  In  this  study,  we  examined  content  validity  or  face  validity 
and  construct  validity.  Cronbach's  alpha  was  used  to  examine  reliability. 

1.     Content  Validation 

Content  validation  is  determined  from  the  content  and  operations  of  the 
measurement  instrument  [Ref.  15:  p.  156].  Others,  such  as  Babbie,  refer  to  content 
validity  as  face  validity  or  logical  validity.  For  instance,  if  we  were  indeed  interested 
in  measuring  client  satisfaction  of  those  clients  supported  by  the  NARDACs,  then 
from  a  logical  point  of  view  each  of  the  items  considered  should  appear  on  its  face  to 
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indicate  the  respondent's  satisfaction  with  the  services  provided  by  the  NARDACs. 
[Ref.  23:  p.  256]  The  content  of  the  measurement  instrument,  therefore,  has  to  be 
designed  to  include  all  important  items  or  attributes  affecting  a  client's  satisfaction 
with  the  NARDAC  support.  As  described  in  Chapter  III,  this  has  been  the 
underlying  theme  of  our  methodology  to  identify  those  items  and  to  test  the  list  of 
items  for  completeness. 

The  items  were  examined,  prior  to  the  pilot,  by  independent  groups  of 
experts  as  the  initial  phase  in  validating  the  completeness  of  the  items  list.  This 
follows  a  similar  approach  used  by  Pearson  in  which  independent  assessments  and 
the  collective  experience  of  expert  groups  provided  different  perspectives  to 
counterbalance  any  significant  omissions  that  could  have  occurred.  [Ref.  16:  p.89] 
The  first  group  of  experts  was  selected  on  the  basis  of  their  experience  in  ADP  and 
interaction  with  the  clients.  This  group  was  composed  of  all  the  NARDAC  Client 
Liaison  Officers  who  reviewed  the  initial  draft  of  the  questionnaire.  The  same 
questionnaire  was  reviewed  by  another  group  of  experts  in  the  academic  environment 
with  research  and  consulting  experience  in  the  fields  of  data  processing,  survey 
research  and  statistics,  and  organizational  management.  This  second  group  was 
comprised  of  faculty  in  the  Information  Systems,  Operations  Research  and 
Administrative  Sciences  curriculums  at  the  Naval  Postgraduate  School.  Based  on  the 
comments  and  recommendations  from  the  first  two  groups  of  experts,  the  item  list 
was  modified  and  the  questionnaire  revised.  The  revised  questionnaire  was  then 
reviewed  by  a  third  group  of  experts  with  extensive  experience  in  the  data  processing 
field  and  management  in  the  Navy  and  with  a  history  of  formal  interaction  with  client 
organizations.  This  last  independent  body  of  reviewers  was  made  up  of  the 
Commanding  Officers  of  the  NARDACs  and  the  NAVDAC  staff,  in  essence,  the 
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NARDAC  Board  of  Directors.  Details  of  these  reviews  are  described  in  Chapter  III, 
Section  D,  Parts  3  and  5.  The  modifications  recommended  by  the  three  review 
groups  and  the  independent  assessment  of  the  completeness  of  the  item  list  indicated 
that  the  important  items  influencing  a  client's  satisfaction  with  NARDAC  services 
had  been  identified  and  validated. 

The  other  aspect  of  content  validity  deals  with  the  logical  soundness  of  the 
operations  for  measuring  the  content  of  the  items.  To  establish  the  operations  for  the 
measurement  of  this  content,  the  Likert  Scale  methodology  was  used.  The  selection 
of  this  methodology  for  appropriateness  and  the  development  of  the  measurement 
instrument  is  described  in  Chapter  III.  On  the  basis  of  this  approach,  we  determined 
that  this  measurement  process  was  further  evidence  of  content  validity,  although 
subjective  in  nature. 

Content  validity  was  likewise  performed  by  the  clients  themselves  during 
the  pilot  phase  of  the  measurement  instrument.  The  clients  participating  in  the  pilot 
were  asked  to  critique  the  questionnaire  by  providing  an  assessment  of  the  following: 

Qarity  of  item  phraseology 

Appearance  of  item  redundancy 

Omission  of  critical  items 

Readability  of  the  survey  and  ease  of  completion 

Clarity  of  the  instructions 
Their  assessment  of  the  above  items  addressed  the  questionnaire's  content  structure 
and  completeness  of  items;  therefore,  further  evidence  of  content  validity  had  been 
achieved. 
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2.     Construct  Validation 

In  the  previous  section,  content  validity  of  the  measurement  instrument  and 
the  process  in  performing  this  test  for  validity  was  described.  Construct  validity 
focuses  on  the  nature  of  the  items  being  measured  and  the  extent  to  which  these 
particular  items  relate  to  one  another.  Thus,  it  attempts  to  measure  the  correlations 
among  many  independent  items  to  determine  whether  these  items  are  strongly  enough 
related  to  describe  a  particular  relational  concept.  In  this  case,  construct  validation 
was  used  to  determine  how  strongly  each  of  the  forty-four  items  related  to  one 
another  and  to  each  of  the  eight  factors.  Factor  analysis  was  the  technique  employed 
in  an  effort  to  perform  this  type  of  validation. 

a.    Factor  Analysis  Description 

Factor  analysis  is  a  multivariate  statistical  tool  that  analyzes 
interrelationships  among  many  items  (e.g.,  questionnaire  responses)  and  then 
explaining  these  items  in  terms  of  their  common  underlying  dimensions  (factors) 
[Ref.  31:  p.  427].  It  is,  in  effect,  a  simultaneously  processed  item  interdependence 
technique  that  tells  which  item  responses  measure  the  same  factor  and  to  what  extent 
they  measure  these  factors. 

In  examining  the  pilot  responses  for  construct  validity,  we  performed 
factor  analysis  on  the  forty-four  item  scores  for  each  of  the  twenty-six  respondents. 
This  was  accomplished  using  the  factor  analysis  software  module  in 
STATGRAPHICS  on  a  Zenith  248  microcomputer  and  the  SPSS-X  factor  analysis 
program  on  an  IBM  3033/4381  computer.  The  purpose  of  these  trials  was  to 
determine  if  there  was  indeed  some  logical  pattern  among  the  forty-four  items 
intercorrelations  and  to  see  what  dimensional  factors  would  be  generated  based  on 
the  intercorrelations  of  these  empirical  input.  STATGRAPHICS  allows  the  use  of 
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the  original  item  responses  and  prompts  for  the  number  of  factors  to  extract  in  the 
analysis.  SPSS-X  performs  similarly  and,  as  an  addition,  provides  the  feature  of 
generating  the  number  of  factors  automatically. 

b.  Factor  Analysis  Procedures 

The  forty-four  item  scores  for  each  of  the  twenty-six  respondents  were 
entered  into  a  file  for  processing  by  the  statistical  software  programs.  In  the  case  of 
STATGRAPHICS,  these  entries  were  simultaneously  stored  by  the  software 
program  to  the  work  diskette.  Parameter  specifications  used  for  the  factor  analysis 
runs  were  Pairwise  (to  handle  N/A  items)  and  Varimax  rotation.  The  output  in 
Pearson  Coefficient  format  (available  on  SPSS-X  only)  contained  more  meaningful 
information  for  evaluation.  The  interested  reader  is  referred  to  the 
STATGRAPHICS  User  Guide  [Ref.  32]  and  SPSS  User  Manual  [Ref.  34]  for  more 
details  on  the  various  factor  analysis  parameter  options. 

c.  Interpretation  of  Factor  Analysis  Output 

The  output  of  a  factor  analysis  program  is  presented  in  matrix  form  and 
consists  of  several  key  components.  The  columns  represent  the  factors  (artificial 
dimensions)  generated  from  the  observed  relations  among  items.  The  values  under 
each  factor  column  represent  the  correlations  between  each  item  and  each  factor  and 
is  referred  to  as  the  factor  loadings.  In  examining  the  output,  one  can  determine  the 
meaning  of  a  given  factor  based  on  those  items  that  load  highly  on  it.  Babbie  points 
out  that  the  generation  of  factors,  however,  has  no  reference  to  the  meaning  of  the 
item,  only  their  empirical  associations.  Furthermore,  he  offers  two  important  criteria 
to  consider  while  evaluating  this  data. 


47 


•  A  factor  must  explain  a  relatively  large  proportion  of  the  variance  found  in  the 
items,  and 

•  Every  factor  should  be  more  or  less  independent  of  every  other  factor.  [Ref. 
23:  p.  328] 

Although  the  interpretation  of  the  complex  interrelationships  found  in 
the  factor  analysis  matrix  output  is  no  simple  matter,  the  following  provides  a  step- 
by-step  method  for  evaluating  the  data.i 


1 .  Examine  the  factor  matrix.  Each  column  denotes  a  separate  factor.  The  values 
beneath  the  columns  are  the  factor  loadings  for  each  item  on  each  factor.  The 
numbers  on  the  left  margin  of  the  matrix  represent  each  of  the  44  items  in  the 
questionnaire. 

2.  Begin  the  analysis  by  starting  at  the  first  item  on  the  first  factor  and  move 
horizontally  from  left  to  right,  looking  for  the  highest  loading  for  that  item  on 
any  factor.  For  sample  sizes  less  than  100,  the  lowest  factor  loading  to  be 
considered  significant  would  be  ±  .50.  If  the  highest  loading  is  significant, 
underline  it. 

3.  Proceed  to  the  second  item  and,  again  moving  from  left  to  right  horizontally, 
look  for  the  highest  loading  for  that  item  on  any  factor  and  underline  it. 
Continue  the  procedure  for  each  item  until  all  the  items  have  been  underlined 
once  for  their  highest  significant  loading  on  a  factor.  Some  items  may  have 
several  loadings. 

4.  Identify  items  that  have  not  been  underlined  (those  that  do  not  load  on  a  factor). 
If  the  item  is  considered  important,  leave  the  item  as  is.  If  the  item(s)  are 
considered  of  minor  importance  to  client  satisfaction,  the  item(s)  may  be 
eliminated  and  derive  another  factor  analysis  solution  with  the  non-loading 
items  eliminated. 

5.  When  all  significant  items  are  loading  on  a  factor  indicating  that  a  factor 
solution  has  been  obtained,  assign  a  name  or  meaning  to  the  pattern  of  factor 
loadings.  Items  with  higher  loadings  are  considered  more  important.  The  final 
result  will  be  a  label  or  a  name  that  represents  each  of  the  derived  factors.  [Ref. 
34:  pp.  250-251] 


1    It  is  recommended  that  the  expertise  of  a  social  researcher  be  used  in  interpreting  the 
factor  analysis  output. 
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d.    Pilot  Results 

The  factor  analysis  execution  runs  of  the  pilot  data  using  both 
STATGRAPHICS  and  SPSS-X  produced  unusable  output  and  therefore 
inconclusive  results.  One  explanation  for  these  results  could  be  the  small  sample  size 
to  item  ratio  for  this  pilot  (.59  :  1).  Hair  et.  al.  emphasizes  that,  as  a  general  rule, 
there  should  be  five  times  as  many  responses  as  there  are  items  (5:1)  to  be  analyzed 
and  that  this  ratio  is  considered  to  be  somewhat  conservative.  He  further  adds  that 
when  dealing  with  smaller  sample  sizes  and  a  lower  sample  size  to  item  ratio,  any 
findings  should  be  interpreted  with  caution.  [Ref.  34:  pp.  250-251]  Ideally  then,  a 
preferred  sample  size  to  ensure  a  more  meaningful  and  empirically  sound  evaluation 
of  the  questionnaire  should  be  at  least  220  (i.e.,  5  times  44  items). 
3.     Reliability  Testing 

According  to  Kerlinger,  "reliability  can  be  defined  as  the  relative  absence  of 
errors  of  measurement  in  a  measuring  instrument".  In  other  words  it  is  the  accuracy 
or  precision  of  a  measuring  instrument.  [Ref.  30:  p.  405]  There  are  basically  two 
types  of  reliability  tests  that  can  be  performed:  test-retest  method,  and  the  internal 
consistency  method  (Cronbach's  alpha).  According  to  Carmines  and  Zeller,  test- 
retest  method  is  one  of  the  easiest  ways  to  estimate  the  reliability  of  empirical 
measurements  [Ref.  31:  p.  37].  In  this  method,  the  same  survey  is  given  to  the  same 
set  of  respondents  after  a  period  of  time,  and  the  correlation  between  the  scores  is 
obtained.  If  exactly  the  same  results  are  obtained  on  the  two  administrations  of  the 
survey,  the  test-retest  reliability  coefficient  will  be  1.00  (i.e.,  perfect  reliability).  The 
problem  with  this  method  is  that  the  respondent's  perception  of  client  satisfaction 
will  presumedly  change  over  time  based  on  the  quality  of  services  provided  by  the 
NARDAC.  Thus,  a  low  test-retest  reliability  correlation  may  not  indicate  that  the 
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reliability  of  the  questionnaire  is  low.  The  test-retest  method  can  also  be  expensive 
and  impractical  to  administer. 

A  much  better  method  of  determining  reliability  is  the  measurement  of  the 
amount  of  error  in  the  instrument.  The  amount  of  error  in  a  measure  can  be 
determined  using  Cronbach's  alpha  test  applied  to  interitem  correlations,  which  can 
be  expressed  as  follows: 

Np' 

a  =  where 

[1  +  P'(N  -  1)] 

N  =  the  number  of  items 

p'  =  the  mean  interitem  correlations. 

The  value  a  will  vary  between  .00  and  1.00,  when  the  mean  interitem 
correlations^  are  between  zero  and  one.  The  value  a  at  1.00  is  perfect  reliability,  but 
a  reliability  score  of  .80  is  considered  acceptable  for  basic  research  [Ref.  15:  p.788]. 
As  with  the  factor  analysis  procedure,  the  reliability  test  was  performed,  but  because 
of  the  small  sample  size,  the  results  were  inconclusive.  It  is  recommended,  however 
that  a  reliability  test  be  performed  on  the  instrument  prior  to  implementing  the  Client 
Satisfaction  survey.  The  ratio  of  sample  size  to  number  of  items  should  be  at  least 
(5:1)  or  greater.  Reliability  testing  is  not  available  in  the  STATGRAPHICS  program 
however;  it  can  be  easily  tested  using  procedure  RELIABILITY  on  SPSS-X.  The 
Alpha  model  (Cronbach's  alpha)  is  the  default  model  in  procedure  RELIABILITY. 


1    To  find  the  mean  interitem  correlation,  the  correlation  coefficients  are  summed  and 
divided  by  the  total  number  of  coefficients. 


50 


D.  REVISED  QUESTIONNAIRE 

The  frequency  distributions  calculated  in  section  B  above  highlighted  two  factors 
that  displayed  an  usually  high  rate  of  not  applicable  responses:  Quality  of  Products 
and  Services  and  Timeliness  of  Services.  Each  item  within  these  two  factors  were 
carefully  examined  and  compared  to  the  profile  of  the  respondents  which  answered 
not  applicable  to  those  items.  From  our  analysis,  we  concluded  that  clients,  such  as 
those  receiving  training  or  Information  Resource  Center  (IRC)^  assistance,  had 
difficulty  answering  the  questions  in  those  two  factors. 

As  a  result,  questions  18  and  28  were  identified  as  being  too  specific  and  were 
revised  to  broaden  their  applicability.  Figure  4.3  illustrates  the  changes  made.  Of 
course,  further  testing  of  the  instrument  with  a  larger  sample  size  will  have  to  be 
performed  to  further  refine  the  instrument,  but  we  feel  these  preliminary  changes  will 
reduce  the  number  of  not  applicable  responses  for  the  factors.  Quality  of  Products 
and  Services  and  Timeliness  of  Services.  See  Appendix  K  for  the  revised 
questionnaire. 


1  The  IRC  at  NARDAC  San  Francisco  was  created  to  help  Navy  commands  and 
other  govemment  agencies  to  deal  with  the  microcomputer  revolution.  Basically,  it  helps 
the  client  take  full  advantage  of  the  productivity  enhancement  capabilities  of  a 
microcomputer. 
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QUALITY  OF  PRODUCT  AND  SERVICES 

(NARDAC's  ability  to  deliver  reliable  services  and  excellent  products) 


CURRENT:     18.  The  quality  of  NARDAC's  data  processing  services. 

REVISED:       18.  The  quality  of  NARDAC's  services. 

(Services  include  training.  Information  Resource  Center  (IRC) 
assistance,  data  processing,  Client  Liaison  services,  studies,  etc.) 


TIMELINESS  OF  SERVICES 

(NARDAC's  ability  to  be  punctual  and  "schedule  conscientious"  with  its  products 
and  services) 


CURRENT:     28.  Timeliness  of  deliverables  from  NARDAC. 

REVISED:       28.  Timeliness  of  services  from  NARDAC. 

(Services  include  training,  Information  Resource  Center  (IRC) 
assistance,  data  processing,  Client  Liaison  services,  studies,  etc.) 


Figure  4.3  Changes  to  items  18  &  28. 
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V.  DATA  ANALYSIS  TECHNIQUES 

A.  INDEX  CONSTRUCTION 

An  index  is  a  composite  measure  very  frequently  used  in  social  research  [Ref. 
27:  p.  361],  Babbie  lists  the  following  advantages  of  an  index: 

•  Allows  a  researcher  to  develop  a  composite  measure  of  variables. 

•  A  single  data  item  might  not  have  enough  categories  to  provide  the  desired 
range  of  variation,  but  an  index  formed  from  several  items  would. 

•  Indexes  are  efficient  data  reduction  devices  :  several  indicators  may  be 
summarized  in  a  single  numeric  score. 

An  index  is  constructed  by  accumulating  scores  assigned  to  individual  attributes. 

The  data  from  the  client  satisfaction  questionnaire  was  formulated  into  an  index, 

which  would  empirically  describe  the  level  of  client  satisfaction.  The  Likert  scale 

was  quantified  by  assigning  values  5,  4,  3,  2,  and  1  to  extremely  satisfied,  satisfied, 

neutral,  dissatisfied,  and  extremely  dissatisfied  respectfully;  the  not  applicable  scores 

were  disregarded  in  constructing  the  index.  According  to  Babbie,  "the  Likert  format 

lends  itself  to  a  rather  straightforward  method  of  index  construction.  Since  identical 

response  categories  are  used  for  several  items  intended  to  measure  a  given  variable, 

each  such  item  can  be  scored  in  a  uniform  manner." [Ref.  27:  p.  375]   Using  the 

values  above,  the  scores  on  the  questionnaire  were  averaged  for  each  factor  to  arrive 

at  a  factor  index: 

m; 


Fj  =  E  L^  Ij  k    where 


Fj  =  individual  factor  index 

mj  =  total  number  of  items  for  factor] 
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L  =  number  of  applicable  responses 
Ijj  =  numeric  score  for  item  k  of  factor  j, 
=  1,  2,  3,  4,  5. 

To  determine  the  overall  Client  Satisfaction  Index  (CSI),  all  of  the  factor  indices  (Fj) 
are  averaged: 

8 


CSI  =  ja    X    ^j     where 


i=l 

N  =  number  of  meaningful  factors 

(i.e.,  factors  where  at  least  one  item  was  applicable) 
i   =  the  number  of  factors. 

The  result  will  be  a  CSI  value  between  one  and  five.  Each  factor  was  given  an  equal 

weight  because  there  were  no  substantiating  reasons  for  differential  weighting  of  the 

factors.  An  interpretation  of  the  index  is  shown  in  Table  5.1. 


TABLE  5.1. 

CSI  RANGE  VALUES 

INTERPRETATION 

4.21  to  5.00 

Extremely  Satisfied 

3.41  to  4.20 

Satisfied 

2.61  to  3.40 

Neutral 

1.81  to  2.20 

Dissatisfied 

1.00  to  1.80 

Extremely  Dissatisfied 

B .  IMPORTANCE-PERFORMANCE  GRID 

The  Importance-Performance  Grid  was  developed  by  Robert  Alloway,  from  the 
Sloan  School  of  Management,  Massachusetts  Institute  of  Technology,  to  help 
identify  which  client  satisfaction  factors  are  most  important  and  to  allocate  resources 
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to  ensure  good  performance  on  those  factors  (i.e.,  "prioritizes  management's 

attention").  [Ref.  29] 

The  last  section  of  the  questionnaire  is  used  to  collect  data  which  can  be  used  to 

develop  an  importance-performance  grid.  The  respondent  is  asked  to  rate  the  relative 

importance  of  each  of  the  eight  performance  related  factors  on  a  scale  of  one  to  five. 

These  importance  scores  are  then  plotted  against  the  average  performance  scores  for 

each  factor  (Fj).    After  all  the  factors  have  been  plotted,  the  grid  is  arbitrarily  divided 

into  four  quadrants.  The  placement  of  the  axes  is  a  managerial  decision  based  on 

what  senior  management  feels  are  achievable  standards  of  success  (e.g.,  see  Figure 

5.1).  Alloway  defines  the  quadrants  as  follows: 

The  upper-right  comer  is  the  relative  success  quadrant.  This  implies 
no  change  is  necessary  in  importance-performance  for  these  criteria. 

The  lower-left  quadrant  is  also  OK  in  the  sense  that  lower  performance 
on  these  comparatively  unimportant  criteria  is  acceptable.  My 
recommendation  for  these  two  quadrants,  leave  them  be,  might  appear 
much  too  benign  until  one  contemplates  the  level  of  managerial 
attention  and  effort  required  to  improve  the  lower-right  quadrant. 

The  lower-right  quadrant  is  the  real  killer.  These  criteria  have  high 
importance  but  low  performance.  These  are  the  criteria  which  ruin  a 
DP  department's  reputation,  drive  users  up  the  wall,  seriously  impair 
DP's  ability  to  deliver,  and  prevent  user  managers  from  receiving  their 
relevant  information. 

The  upper-left  quadrant  should  receive  declining  management 
attention.  Any  increased  efforts  to  improve  performance  here  are  a 
relative  waste  of  resources.  Clearly  DP  management  should  not  steal 
resources  from  the  success  quadrant  for  use  in  the  killer  quadrant, 
rather,  DP  should  reallocate  from  waste  and  OK  to  killer.  [Ref.  29: 
pp.  15-17] 

Each  client  will  have  a  unique  profile  based  on  his  Importance-Performance  Grid 

that  will  focus  management's  attention  on  the  factors  that  the  client  feels  requires 
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greater  attention  by  the  NARDAC.   It  will  also  assist  the  NARDAC  in  managing 
scarce  resources. 
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Figure  5.1  Importance- Performance  Grid  [Ref.  29] 

In  the  above  example  (Figure  4.1),  the  grid  clearly  illustrates  how  the  NARDAC 
performed  on  the  various  factors  and  the  relative  importance  of  each  of  those 
factors^.  Using  AUoway's  definition  of  the  quadrants  as  stated  above,  the  NARDAC 
is  performing  quite  well  in  the  area  of  System  Resources  (2)  and  Staffing  (7),  which 
are  viewed  as  important  to  the  client's  mission.  Therefore,  no  change  in  the  resource 
allocation  is  necessary.  Accessibility  (l)  and  Responsiveness  (6),  however,  rated 


These  scores  are  fictitious  and  were  chosen  for  illustrative  purposes  only. 
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very  poorly  in  performance.  This  rating,  on  face  value,  would  indicate  a  need  by 
management  to  focus  more  attention  in  those  two  areas;  however,  this  particular 
client  views  the  two  factors  as  relatively  unimportant.  Therefore,  it  would  be  more 
productive  to  concentrate  on  the  more  important  factors.  Quality  of  Products  and 
Services  (4)  and  Timeliness  of  Services  (5).  These  two  factors  were  rated  high  in 
importance  but  low  in  performance  and  should  have  management's  highest  attention. 
According  to  Alloway,  resources  should  be  reallocated  in  order  to  bring  factors  in  the 
killer  quadrant  into  the  success  quadrant.  He  suggests  pulling  resources  out  of  the 
waste  quadrant.  In  this  area.  Cost  Management  (3)  and  Timeliness  of  Services  (5) 
rated  high  in  performance  but  low  in  importance,  therefore  efforts  to  improve 
performance  would  be  wasting  scarce  resources. 

C.     FREQUENCY  HISTOGRAM 

Another  interesting  and  informative  way  to  view  the  data  from  the  questionnaire 
is  through  the  use  of  frequency  histograms  (Figure  5.2).  Histograms  are  a  very 
good  and  quick  visual  tool  that  can  be  used  to  see  how  all  the  respondents  answered 
a  particular  item  or  factor.  The  frequency  histogram  fills  in  the  holes  where  the  CSI 
leaves  off.  To  illustrate  this  point,  it  is  plain  to  see  from  Figure  5.2  that  the  clients 
are  quite  satisfied  with  the  accessibility  of  the  NARDAC  with  sixty-five  percent  of 
the  respondents  answering  satisfied  or  higher.  The  histogram  could  also  point  out  a 
poorly  constructed  item  if  there  was  a  very  large  percentage  of  not  applicable 
responses. 
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FREQUENCY  HISTOGRAM  FOR  ACCESSIBILITY 


N/A 


SCORES 


N/A  =  NOT  APPLICABLE 

1  =  EXTREMELY  DISSATISFIED 

2  =  DISSATISFIED 

3  =  NEUTRAL 

4  =  SATISFIED 

5  =  EXTREMELY  SATISFIED 


Figure  5.2  Example  Frequency  Histogram  for  Accessibility 
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VI.     CONCLUSIONS  AND  RECOMMENDATIONS 

A.  CONCLUSIONS 

In  conclusion,  the  research  objectives  of  this  study  were  accomplished.  Client 
satisfaction  was  defined  within  a  domain  of  eight  factors  made  up  of  forty-four 
attributes  (items).  An  instrument  (questionnaire)  was  developed  to  measure  these 
factors  and  an  index  was  constructed  to  derive  an  empirical  measure  of  client 
satisfaction. 

Through  the  process  of  conducting  personal  interviews  with  clients,  holding 
discussions  with  NARDAC  management,  and  digesting  numerous  articles  and 
journals  relating  to  customer  satisfaction,  client  satisfaction  was  adequately  defined 
by  eight  factors.  Forty-four  items  were  developed  to  further  define  the  factors.  A 
self-administered  mail  questionnaire  was  developed  to  collect  the  data.  This  method 
of  data  collection  was  viewed  to  be  the  easiest  and  least  costly,  as  far  as  time  and 
personnel  resources  are  concerned,  to  implement  by  the  NARDACs.  Due  to  time 
and  resource  constraints,  the  survey  instrument  was  piloted  at  one  site,  NARDAC 
San  Francisco,  using  their  fifty-two  clients  as  the  sample.  Twenty-six  respondents 
returned  the  questionnaire,  which  yielded  a  return  rate  of  fifty-seven  percent  after 
accounting  for  undelivered  questionnaires.  Next,  the  pilot  data  was  examined  using 
descriptive  statistics.  As  a  result  of  this  phase  of  the  questionnaire  evaluation,  it  was 
discovered  that  two  factors  exhibited  a  large  number  of  not  applicable  responses.  To 
alleviate  this  problem,  two  items  (18  and  28)  were  revised  to  broaden  their 
applicability.  Content  validity,  or  face  validity,  was  achieved  by  subjecting  the 
questionnaire  to  numerous  critiques.     The  critiques  were  conducted  by  the  nine 
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NARDAC  Client  Liaison  Officers,  faculty  members  at  the  Naval  Postgraduate 
School,  the  nine  NARDAC  Commanding  Officers,  and  by  twenty-six  client 
organizations  during  the  pilot.  Construct  validity  and  reliability  tests  were  also 
performed,  but  due  to  the  small  sample  size  to  item  ratio  (.59  :  1),  the  results  were 
discounted.  Further  testing  with  a  larger  sample  will  have  to  be  conducted  by  the 
NARDACs  prior  to  implementing  the  survey.  Several  methods  of  analyzing  the 
survey  data  were  presented  in  Chapter  V.  An  index  was  developed  to  empirically 
represent  client  satisfaction  and  a  suggested  interpretation  of  the  index  was  provided. 
The  index  is  represented  by  the  following  equations: 

Fj  =  7-   V    Ijif    where 

Fj  =  factor  index  for  factor  j 
mj  =  total  number  of  items  for  factor  j 
L   =  number  of  applicable  responses 
Ijj  =  numeric  score  for  item  k  of  factor  j, 
=  1,  2,  3,  4,  5. 

1     8 
CSI  =  N    X    Pj    where 
i=l 

CSI  =  Client  Satisfaction  Index 
N  =  number  of  meaningful  factors 

(i.e.,  factors  where  at  least  one  item  was  applicable) 
i   =  the  number  of  factors. 

An  Importance-Performance  grid  was  presented  to  help  NARDAC  management 
identify  which  client  satisfaction  factors  are  most  important  to  the  clients  and  how  to 
allocate  resources  accordingly  to  increase  client  satisfaction.  Another  informative 
way  to  view  the  data  collected  from  the  survey  is  to  use  frequency  histograms.   The 
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histograms  can  easily  identify  items  that  are  problematic  (i.e.,  an  item  with  a  high  not 
applicable  response  rate).  As  a  result  of  this  review,  an  item  may  need  to  be  revised 
or  maybe  even  deleted  from  the  questionnaire. 

B.  RECOMMENDATIONS 

In  this  last  section  of  our  study,  we  provide  several  strongly  recommended 
strategies  for  implementing  the  measuring  instrument  that  has  been  developed.  In 
addition,  the  results  and  conclusions  of  our  study  indicate  several  areas  that  may  be 
pursued  for  further  research. 

1.     Implementation  Strategies 

The  following  is  a  list  of  recommendations  for  measuring  and  analyzing 
client  satisfaction  for  the  Navy  Regional  Data  Automation  Centers: 

•  More  rigorous  and  extensive  validation  of  the  measurement  instrument  should 
be  conducted.  With  a  larger  sample  size  (i.e.,  at  least  220  respondents),  the 
measurement  instrument  could  be  fully  tested  and  evaluated  for  construct 
validity  using  the  factor  analysis  method  and  for  reliability  using  Cronbach's 
alpha. 

•  The  results  of  this  study  and  any  follow-on  actions  or  plans  should  be 
published  for  dissemination  to  the  NARDAC  clients.  This  information  could 
help  to  bolster  general  acceptance  of  the  instrument  by  the  client  community.  It 
is  especially  important  that  the  clients  who  participated  in  the  interview  and  the 
pilot  receive  this  feedback  to  show  confidence  that  their  comments  were  used 
constructively. 

•  The  results  of  this  study  should  also  be  provided  to  all  NARDAC  employees 
affected  by  this  study.  Such  information  would  help  to  generate  positive 
support  and  interest  in  this  area  and  foster  general  acceptance  of  the 
measurement  instrument. 
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The  measurement  instrument  should  be  prepared  in  booklet  form  to  present  a 
more  professional  appearance.  The  booklet  is  bound  and  not  loosely  attached 
together.  Moreover,  this  is  the  norm  in  industry  and  professionally  prepared 
surveys. 

Responsibility  for  the  maintenance  and  administration  of  the  measurement 
instrument  and  its  associated  database  should  be  assigned  to  the  Client  Liaison 
Officer  of  each  NARDAC.  This  provides  centralized  administration  of  client 
related  data  and  the  possible  incorporation  of  the  client  satisfaction  index  (CSI) 
into  the  NARDACs  client  profile. 

The  instrument  should  be  initially  administered  to  all  NARDAC  clients  to 
establish  a  baseline  CSI  by  obtaining  100%  participation.  From  our  extensive 
research  and  lessons  learned  from  the  pilot,  the  questionnaire  should  be 
delivered  to  the  clients  in  person  by  a  NARDAC  representative  (e.g.,  the  Qient 
Liaison  Officer/staff).  This  ensures  certainty  of  delivery  and  provides  the 
NARDAC  representative  the  opportunity  to  update  profile  information 
regarding  the  client.  The  return  of  the  questionnaire  may  be  accomplished  by 
mail  using  pre-addressed  envelopes  with  telephone  follow-up,  or  by  collecting 
the  questionnaire  in  person  on  a  predetermined  date.  To  keep  the  CSI  database 
current,  the  survey  should  be  administered  at  least  biannually  and  may  be 
conducted  after  periodic  Advisory  Board  or  Qient  Council  meetings. 

A  cover  letter,  such  as  the  one  in  Appendix  C,  should  be  attached  in  front  of 
the  questionnaire. 

The  survey  results  can  easily  be  processed  and  maintained  on  a 
microcomputer.  A  microcomputer-based  spreadsheet  software  package,  such 
as  LOTUS  1-2-3,  can  be  used  to  consolidate  the  data  from  the  questionnaire 
and  compute  the  client  satisfaction  index.  A  spreadsheet  package  with  graphics 
capabilities  can  produce  visually  enhanced  output  of  the  data  (e.g.,  CSI 
histograms,  frequency  histograms  of  respondents,  or  other  graphics  relating  to 
the  respondents). 

The  measurement  instrument  should  be  used  to  track  client  satisfaction  over 
time  and  circumstance.  With  trend  analysis,  the  effects  of  technological  or 
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policy  changes  implemented  by  the  NARDACs  on  client  satisfaction  could  be 
identified  and  evaluated. 

2.     Further  Research  Areas 

The  following  is  a  list  of  recommendations  for  further  research: 

•  The  forty-four  items  of  the  measurement  instrument  could  be  re-examined. 
Further  examination  of  the  items  could  provide  insight  into  the  characteristics 
of  each  item  which  could  further  define  client  satisfaction.  This  would  be 
beneficial  in  identifying  performance  elements  for  the  NARDACs. 

•  The  use  of  the  measurement  instrument  could  be  incorporated  into  a 
productivity  model  that  would  include  effectiveness  and  efficiency  measures. 
This  would  allow  for  the  setting  of  target  goals  for  each  NARDAC  and  the 
measurement  of  progress  toward  those  goals,  [adapted  from  Ref.  16:  p.  191] 

•  The  measurement  instrument  could  be  tailored  to  other  ADP  service 
organizations  requiring  the  measurement  of  client  satisfaction. 
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APPENDIX   A.    MAJOR  NARDAC  SERVICES 

DATA  PROCESSING  SERVICES 

TIME  SHARING 
DEDICATED 

ANALYSIS  AND  PROGRAMMING  SERVICES 

MICROCOMPUTER 

MINICOMPUTER 

MAINFRAME 

MICROCOMPUTER  TRAINING 

ADP  SECURITY  SERVICES 

TRAINING 

RISK  ASSESSMENT  ANALYSIS 

DATA  COMMUNICATIONS  PLANNING  SERVICES 

ADP  EQUIPMENT  PROCUREMENT  ASSISTANCE  SERVICES 

TECHNICAL  CONSULTATION  SERVICES 
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APPENDIX   B.    CLIENT  ORGANIZATIONS  INTERVIEWED 
SAN  DIEGO: 

COMMANDER,  NAVAL  AIR  FORCE,  U.S.  PACIFIC  FLEET 

COMMANDER,  NAVAL  SURFACE  FORCE,  U.S.  PACIFIC  FLEET 

FLEET  ACCOUNTING  AND  DISBURSING  CENTER,  U.S.  PACIHC  FLEET 

NAVAL  AIR  STATION  MIRAMAR 

NAVAL  AIR  STATION  NORTH  ISLAND 

NAVAL  AVIATION  DEPOT  NORTH  ISLAND 

NAVAL  SEA  SUPPORT  CENTER,  U.S.  PACIFIC  FLEET 

NAVAL  STATION  SAN  DIEGO 

PERSONNEL  SUPPORT  ACTIVITY  SAN  DIEGO 

PERSONNEL  SUPPORT  ACTIVITY  NAVAL  TRAINING  CENTER 

MONTEREY: 

NAVAL  POSTGRADUATE  SCHOOL 

SAN  FRANCISCO: 

COMMANDER,  NAVAL  BASE  SAN  FRANCISCO 

MILITARY  SEALIFT  COMMAND,  U.S.  PACIFIC  FLEET 

NAVAL  AIR  STATION  ALAMEDA 

NAVAL  AVIATION  DEPOT  ALAMEDA 

NAVY  SUPPLY  CENTER  OAKLAND 

PERSONNEL  SUPPORT  ACTIVITY  SAN  FRANCISCO 

SHIP  INTERMEDIATE  MAINTENANCE  ACTIVITY  ALAMEDA 

WASHINGTON,  D.C.   : 

NAVAL  DATA  AUTOMATION  COMMAND 

NAVY  INSPECTOR  GENERAL 

CHIEF  OF  NAVAL  PERSONNEL 

COMPTROLLER  OF  THE  NAVY 

PERSONNEL  SUPPORT  ACTIVITY,  NAVAL  DISTRICT  WASHINGTON 
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NORFOLK: 

COMMANDER  IN  CHIEF,  U.S.  ATLANTIC  FLEET 

FLEET  ACCOUNTING  AND  DISBURSING  CENTER,  U.S.  ATLANTIC  FLEET 

NAVAL  AIR  STATION  NORFOLK 

NAVAL  AVIATION  DEPOT  NORFOLK 

NAVAL  FACILITIES  ENGINEERING  COMMAND 
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APPENDIX  C.    CORRESPONDENCE 

17  August  1987 

Commander 

Naval  Interview  Headquarters 
U.S.  Atlantic  Reet 
Norfolk,  V A    23150 

• 

Dear  Interviewee, 

In  a  continuing  effort  to  better  serve  their  customers,  the  Naval  Data  Automation  Command 
(NAVDAC)  and  the  Navy  Regional  Data  Automation  Centers  (NARDACs)  have 
undertaken  steps  to  improve  customer  satisfaction.  As  part  of  this  effort,  they  have 
requested  our  assistance  in  developing  an  index  which  will  be  used  as  a  measure  of 
customer  satisfaction.  We  are  students  at  the  Naval  Postgraduate  School  in  the  Computer 
Systems  Management  curriculum.  We  are  conducting  this  research  in  partial  fulfillment  of 
our  Master  of  Science  Degree  in  Information  Systems  and  in  hopes  of  providing  NAVDAC 
and  the  NARDACs  with  a  meaningful  and  useftil  tool. 

To  develop  the  customer  satisfaction  index,  we  must  first  identify  those  critical  factors 
which  the  customers  would  use  to  evaluate  the  services  provided  by  the  NARDAC.  As 
discussed  in  our  phone  conversation,  we  will  be  conducting  a  personal  interview  with  you 
regarding  these  customer  satisfaction  attributes  on  the  «date»  of  August  «time».  We  expect 
the  interview  to  last  approximately  one  hour.  Below,  is  a  representative  list  of  factors 
which  we  feel  relate  to  customer  satisfaction.  We  would  appreciate  your  views  on  them 
during  the  interview,  or  any  others  that  you  think  may  be  pertinent. 

Computer  capabilities  (mainframe/mini/micro  support) 

Ease  of  obtaining  services 

Systems  design  and  programming 

Technical  support 

ADP  consultation  and  assistance 

Training 

Cost  of  ADP  services 

Customer  support 

Timeliness  and  quality  of  products  and  services  provided 

If  you  have  any  questions  concerning  the  interview  or  our  schedule,  please  leave  a  message 
for  us  at  our  curriculum  office,  Autovon  878-2174/2175  or  commercial  (408)  646- 
2174/2175,  and  we  will  contact  you  as  soon  as  possible. 

We  appreciate  your  assistance  in  this  effort  and  look  forward  to  meeting  you  on  the  <date>. 

Sincerely, 


Prima  A.  Morris         Robert  J.  Birdwell 
LCDR,  USN  LT,  SC,  USN 
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19  October  1987 


Dear  Client  Liason  Officer, 

As  part  of  the  continuing  effort  to  better  serve  their  customers,  the  NAVDAC  Board  of 
Directors  requested  our  assistance  in  developing  an  index  which  will  be  used  as  a  measure 
of  customer  satisfaction.  •We  are  students  at  the  Naval  Postgraduate  School  in  the 
Computer  Systems  Technology  curriculum.  The  research  we  are  conducting  in  this  area  is 
in  partial  fulfillment  of  our  Masters  of  Science  Degree  in  Information  Systems,  and  will 
hopefully  provide  NAVDAC  and  the  NARDACs  with  a  meaningful  and  useful  tool. 

In  developing  the  customer  satisfaction  index,  we  have  identified  those  critical  factors 
which  existing  customers  would  use  to  evaluate  the  services  provided  by  the  NARDAC. 
We  obtained  these  relevant  data  through  a  series  of  personal  interviews  from  a  cross- 
sampling  of  customers  served  by  NARDAC  San  Diego,  NARDAC  San  Francisco, 
NARDAC  Washington  and  NARDAC  Norfolk.  These  interviews  were  conducted  in  the 
format  shown  in  enclosures  (1)  and  (2).  In  addition  to  the  interviews,  we  have  also 
conducted  comprehensive  literature  searches,  reviewed  customer  satisfaction 
surveys/questionnaires  currently  in  use  by  govemment  and  commercial  organziations,  and 
interviewed  a  number  of  marketing  and  client  liason  representatives  in  private  industry  to 
complement  our  research. 

We  value  your  input  and  field  expeerience  in  the  client  liason  area.  Please  review  the  initial 
draft  of  the  customer  satisfaction  survey  (enclosure  (3))  and  forward  your  comments  to  us 
by  1  November  1987.  Enclosed  is  a  pre-addressed,  stamped  envelope  for  your 
convenience  and  the  timely  delivery  of  your  comments.  The  return  date  is  critical  to  our 
research  deadline;  therefore,  we  would  appreciate  your  cooperation  in  helping  us  meet  our 
milestones. 

If  you  have  any  questions  regarding  the  survey  and/or  our  research,  please  leave  us  a 
message  at  Autovon  878-2174^175  or  commercial  (408)646-2174/2175  and  we  will  return 
your  call  as  soon  as  possible. 

Sincerely, 

Prima  A.  Morris,  LCDR,  USN 

Robert  J.  Birdwell,  LT,  SC,  USN 
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13  November  1987 


Commanding  Officer 
NARDAC 


Dear  Captain, 

We  are  forwarding  two  (2)  drafts  (revised)  of  the  NARDAC  Client  Satisfaction  Survey, 
which  we  developed  in  response  to  requirements  from  the  NARDAC  Board  of  Directors, 
for  your  review  and  comments.  Both  questionnaires  are  identical  in  content  but  differ  in 
format.  As  part  of  our  research,  please  indicate  which  questionnaire  format  (long  form  or 
short  form)  you  prefer  and  why.  As  requested  by  Commander  Taylor,  please  forward 
your  comments  via  electronic  mail  to  NARDAC  Pearl  Harbor  by  23  November  1987. 

If  you  have  questions  regarding  the  questionnaires  and/or  our  research,  we  may  be  reached 
at  Autovon  878-2174/2175  or  commercial  (408)  646-2174/2175. 


Very  Respectfully, 


Prima  A.  Morris,  LCDR,  USN 


Robert  J.  Birdwell,  LT,  SC,  USN 
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7  December  1987 


Commander 

Naval  Client 

U.S.  Pacific  Fleet 

San  Francisco,  CA  94130 


Dear  Client, 


We  are  students  at  the  Naval  Postgraduate  School  in  the  Computer  Systems  Technology 
Curriculum  currently  developing  a  Client  Satisfaction  Index  for  the  Navy  Regional  Data 
Automation  Centers  (NARDACs).  The  Client  Satisfaction  Index  is  designed  to  measure 
each  client's  level  of  satisfaction  with  the  products  and  services  provided  by  the  NARDAC. 
In  order  to  arrive  at  this  measurement,  a  questionnaire  is  used  as  the  instrument  from  which 
the  Client  Satisfaction  Index  is  derived.  This  questionnaire  was  developed  based  on 
comments  from  NARDAC  clients  whom  we  interviewed  in  your  area  and  other 
geographical  locations. 

Please  assist  us  in  validating  the  Client  Satisfaction  Survey  instrument  by  completing  the 
enclosed  questionnaire.  We  also  request  your  personal  critique  of  the  questionnaire. 
Please  comment  on  the  following: 

1.  Are  there  any  items  you  didn't  understand? 

2.  Are  there  any  items  you  felt  were  redundant? 

3.  Are  there  any  items  you  feel  are  critical  to  client  satisfaction  that  were  not 
addressed  in  the  survey? 

4.  Was  the  survey  easy  to  read  and  complete  (format)? 

5.  Were  the  instructions  to  complete  the  survey  sufficient? 

Please  write  your  comments  directly  on  the  survey  below  the  items  you  are  addressing  or  at 
the  end  of  the  survey.  We  are  soliciting  your  personal  comments,  therefore  a  formal  reply 
from  your  command  is  not  required.  Your  comments  and  the  results  of  your  completed 
survey  will  be  held  in  strict  confidence,  to  be  used  only  by  us  to  assess  the  quality  and 
validity  of  the  survey  instrument. 

In  order  for  us  to  meet  our  scheduled  milestones,  please  return  your  completed  survey  in 
the  enclosed,  pre-addressed  envelope  as  soon  as  possible  but  not  later  than  17  December 
1987.  If  you  have  any  questions  regarding  the  survey  or  any  aspect  of  our  research 
efforts,  we  may  be  reached  at  Autovon  878-2174/2175  or  commercial  (408)646- 
2174/2175. 

Thank  you  for  your  interest  and  cooperation. 

Sincerely, 

Prima  A.  Morris,  LCDR,  USN 

Robert  J.  Birdwell,  LT,  SC,  USN 
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«NARDAC  COMMAND  LETTERHEAD» 

Date 

From:  Commanding  Officer,  Navy  Regional  Data  Automation  Center, 


To:       Commanding  Officer, ,  (Code:  XXX) 

Subj:     CLIENT  SATISFACTION  SURVEY 
End:     (1)  Client  Satisfaction  Survey 

1 .  As  part  of  a  continuing  effort  to  improve  our  service,  please  take  a  few  minutes  to 
complete  the  enclosed  questionnaire. 

2.  The  purpose  of  the  Client  Satisfaction  Survey  is  to  assist  us  in  determining  what 
action  can  be  taken  to  develop  more  effective  ADP  support  for  our  customers.  Your 
response  will  enable  us  to  better  understand  your  present  and  future  needs.  The 
questionnaire  is  designed  to  snapshot  present  conditions;  therefore,  please  answer  the 
questionaire  to  reflect  current  conditions. 

3.  We  feel  this  survey  will  be  mutually  beneficial.  We  appreciate  your  continued 
assistance  in  helping  us  serve  you. 
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APPENDIX   D.    INTERVIEW  INFORMATION  SHEET 


DATE  OF  INTERVIEW: 

NAME  OF  ORGANIZATION: 

NAME  OF  INTERVIEWEE: 

GRADE  /  RANK: 

CURRENT  POSITION: 

LENGTH  OF  TIME 
INVOLVED  WITH  NARDAC: 

TYPE  OF  SERVICE(S): 
NARDAC  PROVIDES: 


FREQUENCY  OF  INTERACTION 
WITH  NARDAC  (i.e.,  daily, weekly): 
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APPENDIX  E.    INTERVIEW  QUESTIONS 


1 .      What  are  the  factors  or  areas  you  feel  that  the  NARDAC  customers  should  use 
to  measure  customer  satisfaction? 


2.      What  is,  in  your  opinion,  a  satisfied  NARDAC  customer? 


3 .      Rank  the  factors  in  order  of  importance: 

Computer  capabilities  (mainframe/mini/micro  support) 

Ease  of  obtaining  services 

System  design  and  programming  expertise 

Technical  support 

ADP  consultation  and  assistance 

Training 

Cost  of  ADP  services 

Customer  support 

Timeliness  and  quality  of  products  and  services  provided 


4.      Specifically,  how  would  you  evaluate  or  measure  each  of  the  factors? 
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APPENDIX   F.    ORIGINAL  QUESTIONNAIRE 


1 1   nAmAc  cusTOfjci^  SAVsFAcmN  sumfef 

Pteasa  olaca   an   X  in   tha   bar  which   b»at  d«acrib»a  vour  i«v»l  of   aatmactlon  with   aach  o(  tha  following   statamanH. 
(Additional  cnmments  are  waicoma). 


Customar    Support 

1 .     NAPOAC's  undarstanding  of  tha  imnart  on  your  command  for  lata  or  Inaocurata  products 
or  sarvicas. 


LJ   0    NOT  APPUCABLE 

LJ  5  VEBVSATISBED 

LJ  4  SATTSF1B3 

U  3   NEtTVieR  SATTSFED  NOR  OISSATISFED 

[J  2  OISSATISFIHJ 

LJ  1    VERY  OBSATISPED 


2.     NARDAC's  familiantv  with  your  command's  mission. 


How  satisfiad  ara  you  with  tha  lavel  of  assistanca  orovidad  In  prapanng  proposals  for 
naw  projacts? 


U  0  NorAP«>tXAaL£ 

U  5  VERYSATlSreO 

LJ  4  SATtSFB} 

U  3  r«rTHEn  SATISFCD  NOn  CHSSATISFIED 

LJ  2  OlSSATtsnS] 

U   1  VEHYOISSA-nSFED 

U  0  NOTAppucAaf 

Lj  5  VERYSATISFCD 

LJ  4  SATIStnEO 

U  3   NEITHER  SAT1SFED  NOR  OSSATtSFEO 

U  2  aSSAVSFBi 

U   1    VERV0ISSAT1SFCO 


4.     Raxibllitv  to  satisfactonly  raspond  to  changas  in  your  spacification  raquiramants. 


LJ    0  NOTAPPUCASLE 
U  5  VERYSATTSFED 
U  4  SATtSFED 

LJ  3   NCrT>IERSATISFEDN0R[»SSAT1SFE0 
LJ  2  OSSATISPED 
LJ  1    VERVOSSATISFEO 


5.     CQurtsQus  and  professional  attituda  of  tha  NAROAC  parsonnal  with  whom  you  daaL 


LJ    0  NOTAPPUCAaLE 
LJ  5  VERYSATISFED 
U  4  SATtSFEO 

LJ  3   NEITHER  SATSFCDKOR0ISSAT1SFED 
LJ  2  OISSATBFB} 
Q  1    VERY  DSSATISFED 


Promulgation  of  aanaral  ouroose  information  about  what  is  happanmg  areund  NAROAC 
(1.8..  naw  hardwara,  systam  softwara,  application  softwara,  training,  naw  parsonnal,  ata) 
through  nawtlattars,  atectronic  bullatin  boards,  and  Ctiant  Council  Maatings. 


0  NOTAPPUCAaL£ 

LJ  5  VERYSAT1SFB} 

LJ  4  SATQFB} 

LJ  3  hCrmStSATSFEONOnOISSATISFCD 

U  2  OSSATISFEO 

LJ   1    VERY0SSAT1SF1E0 
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2    NARDACCUSn»eRSATSFACTK3NSURVFr 


7.     Timalinass  of  response  for  a  request  tor  services. 


8.     Lsvel  of  attentron  niven  to  vour  oroiect  and  command. 


LJ    0  NOTAPPUCASlf 
LJ  5  VERrSATlsnED 
LJ  4  SATISFED 

[_]  3   ^CrTHEn  SATSFEO  NOfl  OtSSATISFED 
U  2  OlSSATISFe} 
LJ  1    VERYOeSATSFED 

LJ    0  NOTAPPUCABL£ 
U  5  VERY  SATtSREO 
LJ  4  SAT1SFCD 

U  3   rCmEn  SATSFEO  NOR  nSSATISFEO 
LJ  2  nSSATISFE} 
LJ  1    VERYOtSSATBFEO 


When  a  problem  assoaated  with  a  NARDAC  product  or  servica  is  reported,  how  satisfied 
are  you  with  the  timeliness  and  Quality  of  the  resolulion? 


LJ    0  NOTAPfUCABLE 
U  5  VEHVSATISFED 
U  4  SATISFED 

U  3   «rT>«R  SATISFED  NOR  nSSATISFED 
LJ  2  nSSATISFED 
U  1    VERY  nSSATISFEO 


Service    Level    Mar^aQement■ 

10.  Aeeuraev  of  the  initial  planning/cost  estimates  for  your  project  request 


0  NOTAPPIXAaE 

LJ  5  VERYSATBFED 

LJ  4  SATISFED 

LJ  3   fCrrVIEH  SATISFED  NOR  DISSATISFED 

LJ  2  CXSSATISFED 

Lj  1    VERYDSSATSFED 


1 1 .  Ability  to  adequately  explain  costs  and  verifv  eharoes. 


1 2.  Format  and  oualitv  of  the  Chargebad<  Report 


[j    0  NOTAPPUOaLE 
LJ  5  VERY  SATISFED 
Q  4  SATSFED 

LJ  3   rCITVIER  SATISFED  NOR  0ISSAT1SFE0 
LJ  2  OISSATISFE) 
U  1    VERYOtSSATSFED 

LJ    0  NOrAPPUCABLE 
Lj  5  VERYSATSFED 
LJ  4  SATISFB3 

U  3  fCnVER  SATISFED  NOR  DISSAT1SFED 
LJ  2  DISSATSFED 
LJ  1    VERYOBSATISFEO 
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1 3    HARDACCUSTOKeRSATSFACTKM  SURVEY 


1 3.  CuiTGnt  ratas  NARDAC  chargas  for  sarvicss  ara  compatitrva  with  industry. 


14.  How  satlsfiad  are  you  with  the  coordination  and  follow-uo  on  ta5ks/pro|acts  assigned  to 
subcontraaors? 


D    0  NOTAPPUCASLE 
LJ  5  HlGH.YAGnEE 
U  4  AGflEE 

LJ  3   NETTICR  AGREE  NOnOISAGRS 
LJ  2  OSAGP£E 
LJ    1    HIGHLY  DISAGREE 

LJ    0  NOTAPPtXAaif 

LJ  5  VERYSATBFED 

LJ  4  SATBFE) 

LJ  3   ^ErT>CR  SATISFCDNOROISSA'nSFCO 

LJ  2    DSSATSFED 

LJ  1     VERYIItSSA'nSFIED 


Assesslbllltv 

15.  Communieation  rhannals  between  you  and  your  NAROAC  point  of  contact. 


LJ    0   NOTAPPUCAaLE 

LJ  5  VEBYSATBFED 

LJ  4  SATSFB} 

LJ  3   NEm^ERSATSFEDNOROSSATIsrED 

LJ  2  OtSSATISFEO 

LJ   1    VERY  OBSATBFED 


16.  PfQceduras  lor  requesting  various  NAROAC  services. 


17.  Hours  of  operations. 


18.  Physical  location  of  NARDAC. 


LJ  0  NOTAppixAaf 

LJ  5  VERYSAT1SFE3 

LJ  4  SATISFED 

LJ  3   KCrmER  SATISFEO  NOn  DISSAT1SFED 

LJ  2   aSSATISFED 

LJ   1    VERYOSSATISFCD 

LJ    0  NOTAPPUCAOLE 
LJ  5  VERYSATISFEO 
LJ  4  SATISFED 

LJ  3   NEtTXER  SATISFED  NOR  0ISSAT1SFEO 
U  2  DISSAT1SFED 
LJ  1    VERYOeSATISFEO 

LJ    0  NOTAPPUCABLE 

LJ  5  VERY  SATISFED 

LJ  4  SATBFED 

LJ  3   NEnXERSATSFEONOnOtSSATISFED 

U  2  OISSATBFED 

LJ  1    VERY  DCSSATBFED 
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|4    HAKMC  CUSTOMER  SATSFACTON  SURVEY 


19.  Vistor  parking  soacas  at  NARDAC. 


U    0  NOTAPPUCABlf 
LJ  5  VERYSATISFED 
U  4  SATSFED 

lJ  3  »crT>«RSA'nsFEONonoissA'nsFED 

LJ  2  OlSSA-nsFED 

U  1   VERV06SATBPED 


Hardware.    Softwara.    and    Communication   Tachnoloov. 

20.  Current  eomputar  hardware  caoabilitias  fmaintrama.  mincomputars.  microcomputars). 


0  NOTAPPUCABLf 

U  5  VERYSATBFED 

LJ  4  UTtSFED 

U  3   hCrTHER  SATSFED  NOR  DISSAT1SFED 

U  2  nSSATSFB} 

LJ   1    VERYOBSATBFED 


21 .  Ability  to  meet  your  proieciad  automated  information  system  requirements. 


0  NOTAPPUCAai 

[J  5  VEPtySATBFED 

U  4  SAIHiHJJ 

U  3   rCITVtEn  SATSFED  NOR  OtSSATISFEO 

D  2  OISSATSFB} 

LJ  1    VERYOeSATSFED 


22.  Currency  and  variety  of  softwara  packages  available. 


23.  Telecommunication  capabilities  and  services  saliafv/meet  mv  command's  ooeraional 
needs. 


U    0  NOT  APPUCABLE 
U  5  VERY  SATSFED 
U  4  SATSFB) 

U  3  NETTVERSATISFEONOROISSATSFED 
U  2  OISSATSFB} 
U   1   VERY06SATSFED 

[J    0  NOT  APftXABl£ 
LJ  5  HKjHLY  AGREE 

U  4  Acre 

LJ  3  hCnVER  AGREE  NOR  OSAGRS 

Q  2  06AGRE 

LJ  1    MQH.YOISAaREE 


Tecfinlcal    Proficiency. 

24.  Level  of  expertise  of  the  technical  staff. 


[j    0  NOTAPPUCAaE 
LJ  5  VERY  SATSFED 
LJ  4  SATSFED 

LJ  3   KEtT>CR  SATSFED  NOR  0ISSAT1SFE0 
U  2  DISSATBFED 
Q  1    VERYOSSATSFED 
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1 5    mRDACajSTOt^RSATSFACTCN  SURVEY 


25.  The  tachnical  staff  is  wall  vars«d  In  a  broad  ranna  of  ADP  subfact  araas. 


0  NOT-APPUO&E 

l_l  5  HIGK.Y  AGREE 

Q  4  tCPE£ 

U  3   NErrVCRAGflEE  NOR  DISAGREE 

LH  2  OSAGRS 

Lj  1  hch.yoism:ree 


26.  Tha  tachnical  staff  Is  kaaping  abraast  of  tha  latast  davelopfnants  in  tha  AOP  arana.  LJ 


0  NarAPf>ixAai£ 

U  5  HIGH.YAGRS 

U  4  AGREE 

U  3  rCmCR  AGREE  NOR  OCSAGRS 

LJ  2  nSAGRS 

LJ  1    HIGH.Y  DISAGREE 


27.  Ability  to  design  and  davalop  craativa  and  innovaliva  svstams. 


LJ    0   NOTAPPUCASU 

LJ  5  VERYSATISPED 

U  4  SATSFED 

LJ  3   fCrrVERSA'nSFEDNOROISSA'nsFEO 

LJ  2  nSSATSFCD 

LJ  1    VERYDSSA-nSFEO 


28.  Parsonnal  assigned  to  your  proiact  damonstratad  tha  propar  sh 
parform  ttia  task(s)  spaafiad  in  ttia  Statamant  of  Work. 


i  laval  raquirad  to 


LJ    0  NOTAPPUCAaE 
LJ  5  VBffSAVSFBi 
LJ  4  SATSHB} 

LJ  3  »CIT>CRSATBFE0N0R0ISSAT1SFED 
U  2  OISSATISFE} 
[j  1    VERYOeSA'nSFEO 


29.  Ability  of  tha  tachnical  staff  to  eommunieate  In  customar  tarms/languaga. 


0  NOTAPPUCABU 

LJ  5  VERYSAT1SFCD 

U  4  SATGFED 

LJ  3   ^CITHER  SATGFED  NOR  DISSATISFED 

D  2  DISSATGFED 

LJ  1    VERYOSSATISFED 


Quality   of    Produeta   and    Sarvlcea. 

30.  Tha  Quality  and  accuracy  of  dalivarablas. 


[J    0  NOTAPPUCABLE 

LJ  5  VERY  SATGFED 

U  4  SATGFB) 

LJ  3   KEfTICRSATISFED  NOR  DISSATGFED 

D  2  OISSATSFED 

LJ  1    VEnYOeSATGFED 
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6     NARDAC  CUSTOUER  SATSFACT70N  SUPVeY 


31 .  Availabllrtv  of  online  and  batch  systams  (i.a.,  tha  total  tima  tha  systam  is  up). 


[J    0   NOTAPPUCAaLE 
lJ  5  VEBY  SATISFIED 

U  4  SA-nsnEO 

Li  3   NEfTHER  SATtSFED  NOR  DCSSATISFED 

LJ  2  OISSATSFED 

U  1    VEPYOBSATTSFED 


32.  Qualitv  of  systam  documantation/manuals  (a.g.,  raadabia,  corract.  up-to-dota). 


0  NOTAPPUCABLE 

U  5  VEBVSATBFED 

LJ  4  SATtSFED 

LJ  3   ^C(THEnSAT1SnEDN0R0ISSAT1SFCD 

U  2  OISSATISFED 

LJ  1    VERYOtSSATSFED 


33.  The  oerformanca  of  your  systam(s)  currently  running  on  NAROAC  computers. 


U    0  NOTAPPUCAa^ 
U  5  VEBYSATBFCD 
LJ  4  SATISFB) 

LJ  3   NEtTHEn  SATSFCD  NOR  OISSATISFED 
LJ  2  OISSATISFED 
LJ   1    VERY  06SAT1SFEO 


34.  NAROAC  develops  reliable  and  cost  effective  e^icatnn  programs  for  their  customare.       LJ 


0  NOTAPPUCABtf 

U  5  HCH-YAGREE 

U  4  AGRS 

LJ  3  NETTVCR  AGREE  NOnOSACREE 

U  2  OBAGRS 

U  1    HIG»«.YOISACnEE 


35.  Response  lime  of  online  svstems. 


0  NOTAPPUCAaf 

LJ  S  VERYSATISFED 

U  4  SATBFED 

LJ  3   NEfTHER  SATBFED  NOR  OISSATISFED 

LJ  2  0ISSAT1SFE) 

O  1    VERYDSSATISFEO 


36.  Maintenanea  performed  on  currant  system*  (La.,  raaponsivanass  in  incorponting 
modifications  and  corrections). 


0  NOTAPPUCABLE 

LJ  5  VERYSATBFED 

U  4  SATBFED 

LJ  3  NEflVCR  SATBFED  NOR  OISSATISFED 

U  2  OISSATBFB} 

U   1    VERYOSSATBFED 
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I?    HARDAC CUSrOtueR SATISPACTXM SUavEY 

37.  Quality  of  training. 


0  NOT  APPUCABUE 

LJ  5  VERY  s»'ns»=CD 

LJ  4  SAIBt-fcU 

[_1  3   NEITMCT  SATtSFED  NOB  DISSATISFED 

LJ  2  nSSATISFED 

LJ   1    VERY  OBSATSFED 


Timeliness    of    Products    and    Servlcta. 

38.  Adharanca  to  oroiect  sehgdules  as  spacifiad  in  ttia  Statemant  of  Work. 


39.  Adherance  to  production  orocassin;!  schedulas. 


40.  Timaliness  of  EiaaiflSilflBflOS- 


41 .  Timetv  notification  of  any  dalays  in  output. 


42.  RaliaWa  and  timely  delivery  of  output. 


LJ    0  NOTAPPtXAaLE 

U  5  VERYSATSFEO 

LJ  4  s«-nsFCD 

LJ  3   KCrTHEnSATtSFEONORIDISSA'nSFED 

U  2  nSSATISFCD 

LJ  1    VERY  OeSATtSFED 

LJ    0  NOTAPPIXABLE 

U  5  VERYSATBFHD 

LJ  4  SAIKI-ED 

LJ  3   NEfTHERSATISFEDNOnaSSATISFED 

LJ  2   nSSATlSFED 

LJ  1    VERY  06SAT1SPED 

LJ    0  NOfAPPUCAaX 

U  5  VERYSATISRED 

D  4  SAIBtf-fcL) 

LJ  3   NEtTVERSATISFEONOROISSATISFEO 

LJ  2  OSSATISFED 

LJ  1    VERY  OSSATSFED 

LJ    0  NOTAPPIXASLE 
LJ  5  HCH-YAGREE 
U  4  AGREE 

LJ  3  rCTT>€R  AGREE  NOnOSAGREE 
U  2  IXSAGRS 
Li  1    HIQH.Y  DISAGREE 

LJ    0  NOTAPPUCAS1£ 
LJ  S  HBH-YAGREE 
U  4  AGREE 

U  3  NEmCR  AGREE  NOR  OSAQREE 
Q  2  06AGREE 
LJ  1    HKM.Y  DISAGREE 


(Opdonal):  r4ama^ 


_Taiiphone^ 


PLEASE  PRINT 

Thank  you  for  complating  this  survey.  The  endosed  envak)pe  has  bean  provided  for  your  convenience. 
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APPENDIX    G.  QUESTIONNAIRE  SECOND  DRAFT 

SHORT  FORM 


HAROAC  CUetfT  SATtSfACTIOM  SVItVgr 

PLEASE  PLACE  AN  X  IN  THE  BOX  WHICH  BEST  DESCRIBES  YOUR   L£VeL   OP  SATISFACTION     WITH  EACH  OF  THE 
FOLLOWING  ITEMS.  {Additional  comments  may  be  made  on  the  last  page). 

USE  THE  FOLLOWING  SCALE: 

Ult^   NOT  APPt.lCAai£       5  VERY  SATISFIED        4  SATTSRED       3   ^erTV^6R  SATSFED  NOB  OBSATISBBD     2  OSSATTSFIED        1    VERY  DISSATISFIED 


AccBssibllltv. 

1 .      How  satisfiad  ara  you  with  tha  availabilitv  at  your  NAROAC  point  of  contaa. 


2.       Th«  omeadurag  for  obtaining  vanou*  products  and  mtvk 
(refw  to  NAVOACINST  5230.1) 


3.      NARDACs  currant  hours  of  ooaraion. 


4.      The  Physical  location  of  NAROAC  ralativ*  to  your  command. 


5.       Tha  Quantity  o<  visitor  oarying  soaeaa  at  NAROAC. 


Svatam     Haaoureaa. 

(Encompasses  harawan.  soltwan.  and  tatapmcuasmg  tachnoloqy) 

S.       How  salisfiad  ara  you  withNAROACs  currant  eomouter  haftiwaf  C^abilitiaa- 
(i.»..  mainframa,  minKomputars.  mcmcomputara} 


7.      NAROAC  ability  to  maat  your  nmiartad  autnmatad  intormadon  systam  raquiramants. 
3.       Tha  currancv  and  vanetv  of  sottwara  packagaa  availabla. 

9.  NAROAC'3  ability  to  previda  talaorocasaino  san/icaa  to  maat  your 
command's  oparatnnal  naads. 

Coat    Manaaament. 

1 0.  How  satisfiad  ara  you  with  NAROACs  adharanoa  to  oroiati  budaata? 

1 1 .  NAROACs  ability  to  adaquataiy  explain  coat  and  vanfv  charoaa. 

12.  Tha  loniiat  of  tha  Chaigabacfc  Raport  ^i*,,  Msy  ID  nMtfwitfu/K/arstand^ 

13.  Tha  aeeuraey  of  tha  Chawyabaek  Raooit 

1 4.  Tha  ciirrant  rata*  NAROAC  chargaa  for  producta  and 


n/aD  sD  4n  3n  20  iD 

n/aD  n  u  n  n  n 

n/aD  n  n  n  n  n 

n/aD  n  n  £1  n  u 

N/An  sD  40  oQ  aD  iD 

n/aO  £2  £1  £2  €i  U 

n/aD  n  n  n  n  a 

n/aO  sD  4n  £3  20  iD 


'aO         sD     40     30     2O      lO 


Wi 


n/aD  sD  40  £3  £]  iO. 

n/aO  sD  4O  3O  2O   iO 

n/aD  sO  4O  3O  20  iO 

m^aD  sD   4O    3O  2O    iO 

n/aO  sO  4O  £1  2D  iD 
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I       NAROAC  CUEHT  SAWACrnOH  SVRVir 
USE  THE  FOLLOWING  SCALE: 
N/A   NOT  APPLCABLf      SVERY  SATISFIED         4  SATSFED       3  NEIT>ER  SADSFCD  NOA  OSSATISFEO     2   OSSATISfnED        1   VERY  OSSATSFCO 

15.     Flaxibilrty  in  Charging  tor  s«rvicas  (e.g.,  fix«d-prca  oomraa.  tlsrad  rata  structura,  ate).  N/aLJ        sLJ     4LJ     sLJ     zLJ     iLJ 


Quality    of    Pfodueta    and    Sarvleaa. 

(N ARC  AC'S  ability  to  dQlivar  nUaDt»  sarvic»s  and  »xc«ll»nt  pmduco) 

1 6.     How  saiisfiad  ara  you  with  ttia  quality  of  hJAROAC  <inftwar»  nmdiiny;? 


1 7.     Tha  quality  of  NAROAC's  data  orocaggina  servicfls. 


18.    Tha  eentrols  ovar  data  sacuritv  . 


1 9.    The  reliability  of  onlina  and  batch  systams  (i.e.,  th»  tolaJ  time  th*  system  is  up). 


20.    Tha  accuracy  of  systam  documantation/manuala. 


21 .     Tha  partormancQ  of  your  sysiam(s)  currantly  running  on  NAROAC  computara. 


22.     The  response  time  of  NAROACs  onlina  systams. 


23.    Whan  a  problam  assooatad  with  a  NAROAC  product  or  saivca  is  raortad,  how  sMisflad. 
ara  you  with  tha  oualrtv  ol  the  fesoluiion? 


Tlmailnaaa    of    Sarvleaa. 


24.  l-low  satisfied  ara  you  with  NAROACs  adharanca  to  nroiect  schedules  as  spadfiad  in  tha 
Suiamant  of  Work? 

25.  Adharanca  to  production  orocassirw  schaduiaa. 


26.    Timalinass  of  progress  reports. 


27.  Software  maiptananee  oarlormad  on  existing  systams  (L:,  msponsiventt  in 
eorroamg  program  arwn). 

28.  Timplinafis  nntjfitation  of  delays  in  output 


29.     Timeliness  of  rialivenihle!t 


30.     Timelineaa  of  rasponsa  for  a  raoueat  for  servieaa. 


n/aD  n  n  n  n  n 

N/aD  sD     40     30     20  iD 

n/aO  n  n  £1  n  n 

N/aO  sO  40  3O  20  iD 

N/aO  sD  4D  30  2O  iD 

n/aO  sO    4O    3O    2O  iD 

n/aO  n  n  n  n  n 

n/aO  sO    4O    3O    2O  iO 

n/aO  sD  4O  3O  2O  iO 

n/aO  sD  4O  3O  2O  iO 

n/aO  sO  4O  3O  2O  iO 

N/aO  sD  40  3O  2O  iO 

n/aD  sO  4O  3O  2O  iD 

n/aO  sO  4O  3O  2O  iO 


N/< 


■aO     £]  4O  sO  2O  iO 
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NAROAC  CUEffT  SATISfACTIOH  SURVrf 
USE  THE  FOLLOWING  SCAL£: 
f4/A  NOT  APPUCASLE      S  VERY  SAT1SFED         4  SATBFEO       3  NErT>CR  SATISFED  NOR  OSSATISnEO     2   OSSATISFEO        1    VERY  OISSATSFED 

31.  Whan  a  probl«m  associat»d  With  a  NAROAC  product  or  sarvcs  «  raponad.  how  sMofwd  N/aLJ         SLJ     4-J     3lJ     zIJ     iU 

are  you  wrth  tha  timeliness  ot  the  fasolution?. 

Comoatenev 

32.  How  satisfiad  ara  you  with  tha  tachnical  staff's  level  of  exoenise?.  N/aLJ        SLJ     4LJ     sLj     zLJ     iLJ 
(Tectinical  satff  mdudas  oomputar  spaaaJists,  programmars.  and  systam  anaJysts) 

33.  Tha  technical  staffs  ranoe  of  knowledge  in  thair  raspactiva  AOP  fialds..  N/aLJ        SLJ     4LJ     SLJ     2LJ     iLJ 

34.  Tha  tachnicai  staffs  knowledge  of  the  latest  davaleomems  in  thair  N/aO         SLJ     4O     3LJ     2LJ     iLJ 
raspactiva  AOP  fialds. 

35.  NARDACs  ability  to  datign  and  davalop  creative  and  mnovativa  systams.  N//0        SLJ     4LJ     3LJ     2O     1LJ 

36.  ThaNARDACparsonnalassignadtoyourpTOiact  O-a.-dothay  damonstratattta  N/ALJ        sO     4LJ     £2     2LJ     lU 
□rooer  skill  level  to  partorm  tha  task(s)  spacifwd  in  tha  Statamant  of  Work. 

Coordination    and    Communleatlona 

37.  How  satisfied  ara  you  with  NARDACs  undarstanding  of  ttM  tmBacLon  your  command  tor  N/aO        SLJ     4J_J     a_J     2LJ     lLJ 
laie  or  inaccurate  oroOucts  or  services? 

38.  NARDAC-8  familiantv  with  vour  command's  mission  N/aD  sD  40  30  zD  iD 

39.  Tha  level  of  assistance  oiovidad  in  oraDamq  orooosals  tor  naw  oroiacla  N/aLJ  sLJ  4LJ  3LJ  2Lj  lLJ 

40.  Flexibilltv  to  satisfactorily  raapond  to  changaa  in  your  spac>fic«tion  raquiramant*.  N/ALJ  SLJ  4lJ  3LJ  2LJ  lLJ 

41.  Courteous  and  oroteasional  attituda  of  tha  NARDAC  panennai  with  whom  you  daaL  N/aLJ  sQ  4LJ  3LJ  2lJ  iLJ 

42.  Promuloation  of  oaneral  ouroosa  Informiition  ahoul  wht  i«  hanoaninn  found  NARDAC  N/aLJ         SLJ     4Lj     sLJ     2LJ      lLJ 
I'l.a..  naw  harOwara.  systam  sottwara.  application  sottwara,  traming,  naw  parsonnal.  ale) 

through  nawslattars,  atadronic  buUatin  boards,  and  Client  Counai  maaungs. 

43.  Tha  coordinaton  and  follow-up  on  tasks/oroiaets  aasionad  to  subeomractors.  N/aLJ         SLJ     4LJ     3LJ     2LJ     iLJ 

44.  Tha  ability  of  NARDACs  tachnkal  alaff  to  communicate  in  diem  terms/lanouaoa.  N/aO        sO     4O     sLj     2LJ     iLJ 
(Tachnicai  satff  indudaa  oomputar  spacmlists,  pmgrmmman.  and  systam  analysts) 
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I       NAROAC  CUEHT  SATtVACnOH  SUnVFY 

USE  THE  F0LL0W1NO  SCAL£: 

r*A   NOT  APPLCABLE      5  VERY  SATISRED         4  SATBFCD       3  ^etT^€R  SAVSFfED  NOfl  CUSSATISFED     2   OSSATISFHD        1    VERY  OISSATBFIED 


PLEASE  PLACE  AN  X  IN  THE  BOX  WHICH  BEST  DESCRIBES  THE  RELATtVE    IMPORTANCE  TO    YOUR 
ORGANIZATION   OF   EACH   OF  THE   FOLLOWING    PERFORMANCE    CRITERIA    AS   IT   RELATES  TO  CUENT 
SATISFACTION. 


USE  THE  FOLLOWINO  SCALE: 


1.      Aceasslblllty 


2.      Syatam    R«soure«s 


3.      Cost    Managamant 


4.      Quality  of   Producta  and   Sarvleaa 


5.      Tlmallnaaa   of   Sarvleaa 


6.      Compatancy 


7       Coordination   and   Communlcatlona 


5  CRfTICAL  4  IMPORTANT         3  AVERAGE         2  NOT  MPOHTANT        1  IRRELEVANT 


n 
n 
n 

sD 

n 
n 


n 

n 
n 

n 


n 
n 
n 
n 
n 
n 
n 


n 
n 
n 
n 
n 
n 
n 


n 
n 
n 

n 
n 
n 


Commanta: 


N>T»_ 


_CddB_ 


.Tstaphona^ 


nSASEFRNT 

Thank  you  for  completing  this  survey.  The  enclosed  envelope  has  been  provided  for  your  convenience. 
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APPENDIX   H.  QUESTIONNAIRE  SECOND  DRAFT 

LONG  FORM 


I   nAmtc  CLENT SATJSFAcnoN  suaveY 
Please  place  on  X  in  the  box  which  best  describes  your  Inrl  nf  mnrffirnnn  with  each  of  the  following  qjti-mymf 

{Addiiionai  commenu  art  weicoma). 


Aceeasibllltv 

1 .     How  saosfiad  ar*  you  with  th«  availahiliiv  of  your  NAROAC  point  of  contact? 


2.     Th«  orocaduraa  for  otnaintng  vanous  preducts  and  sarvica*. 
(ntor to  NAVDACINST  5230. 1) 


LJ    0  HJTAffKJUaiM 
LJ  5  VERVSATISFB} 
U  4  SATBFED 

l_I  3   NErrVCR  SATSFCO  NOR  OrSSATISFlEO 
Cj  2  OSSATISFEO 
LJ  1   VERV  OSSATtSFCO 

LJ  0  NorA»>nxAa£ 

LJ   5   VStVSATBFB} 

LJ  4  SAT1SFC0 

Lj  3   rCnvCR  SATBFEO  NOR  OISSATISFEO 

LJ  2  OBSATISFED 

LJ  1   VERYOaSATOFB) 


3.     NAROACs  currant  hours  of  ooafaiion. 


0  rcn-APnXASLE 

U  S  VERVSATISFE} 

LJ  4  SATBRH) 

LJ  3  NEirVCR  3ATBFC0  NOR  OISSATISREO 

LJ  2  OSSATISFEO 

lJ  1   VERV0BSAT1SFED 


4.     Th«  ortvsical  iQcaiion  of  NAROAC  ralanv*  to  your  command. 


S.     Tha  quantity  of  viator  oartdno  snacaa  at  NAROAC. 


System     Heaoureaa. 

(Ericompaaaaa  tmnfumn,  sottwrnrm,  and  t»i»pmc»saing  tachnology) 

8.     HOM  satisflad  ara  you  witfi  NAROACs  currant  computar  harrtwant  rarattSttan. 
(l.a..mMintnmm,  fntntoomputarx  mierooompune$f  7 


[j    0  NOTACnjCMLE 
LJ  5  VERVSATISFB} 
LJ  4  SATBTCO 

[J  3  ^CITVERSATBFEONOR0ISSATISF1E0 
LJ  2  OSSA-RSPED 
LJ  1   VERV0SSAT1SFE] 

LJ    0  NOrAPPUCASLE 
LJ  S  VB1VSATBFE0 
LJ  4  SAHSHB) 

LJ  3  FCITTCRSATBFCDNOR0ISSAT1SFED 
LJ  2  OSSATBFCD 
LJ  1   VERYOBSATISFeD 


0  Nor/tf>nxAaL£ 

LJ  S  VERVSATBFCD 

U  4  SATBFED 

[J  3  ^cmCRSATBFCDN0R0ISSAT1SFC0 

LJ  2  OBSATOnB] 

LJ  1   VERVOSSA-nsFED 


88 


I     NAmACCLCh/TSATISFACmONSUftVrf 

7.     NAROAC's  ability  to  maat  your  proiaciad  automatad  inlormation  systam  raquiramams. 


0   NOTM*>UCAa£ 

LJ  5  VEFVSA-nSFE) 

LJ  4    SATBFED 

LJ  3   ^CrT>CR  SATSFEI}  NOR  0ISSAT1SFE0 

LJ  2  DBSATtSPED 

LJ   1    VERYDBSATSFB) 


8.     The  njiranev  and  vanatv  of  software  packagas  availabla. 


9.     NARDAC's  abilitv  to  ofovida  tBtBPmeassrno  seryieas  to  maai  vouf  eommand'a 
oparational  naads. 


U    0  NOTAPRXAOf 
LJ  5  VERVSATBFED 
LJ  4  SATBRED 

LJ  3   rcrrvCR  SATBFED  NOR  DISSATI5FED 
LJ  2  DBSATBFCD 
LJ  1   VERVDSSA-nSFE} 

LJ    0  NOTAPRXMLE 
LJ  5  VERYSATBFE) 
LJ  4  SATISFCO 

LJ  3  OCrrVCR  SATBFEO  NOR  0ISSAT1SFE0 
LJ  2  OBSATSnED 
LJ  1   ^RirDSSA-nSFED 


Cost    Management. 

1 0.  How  satsfiad  ara  you  with  NARDAC's  adharancs  to  ofoiact  bufloata? 


LJ   0  NtTrAPnjCMLE 

LJ  S  VERVSATBFED 

LJ  4  SATBFED 

LJ  3  NErTHERSATBFEONOROISSATISFEO 

U  2  OBSATISPEO 

LJ  1   VERV06SATBFEO 


11.  NARDACs  abilitv  to  adaoualatv  enolain  costs  and  vartfv  ehar^aa. 


0  N0T4 

LJ  5  VERVSATOFED 

LJ  4  SATBFCD 

LJ  3   >CtT>CRSATBFEDNOR0ISS*'nsFCD 

LJ  2  DeSATBFCD 

LJ  1   VBtVOBSA'nSFED 


1 2.  Tha  format  of  tha  Chargaback  Raport  fl.»..  tmsy  to  nmt  and  untitrsandi. 


0  NOTAPPLCMLE 

LJ  5  VERTSATOFE} 

LJ  4  SATBFED 

LJ  3  ^fTVCRSATSFEDNOROISSATtSFED 

LJ  2  DBSATBRED 

LJ  1   VERVDBSATBFe) 
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NAPDAC  CUENT SATKf  ACTION  SURVTf 

13.  Tha  accuracy  o(  Iha  Charqotuck  Raoort. 


14.  Tha  eufTBnt  rates  NARDAC  eharana  tor  modueta  and  atviCM. 


15.  Rambiirtv  jn  chatying  for  satvicas  (•.g.,  flx«d-phc»  comma,  tartd  rm»  struaun,  wtc). 


lJ    0  NOTAPnXMLE 
Lj  5  VERYSATBFHD 
LJ  4  SATSFEO 

Lj  3   NEnVERSATBFEDNOROISSA-nSFCD 
LJ  2  DSSAIBFCO 
LJ  1   VERYOSSATISFED 

[J    0  NOTAPnXMLE 
LJ  5  VERYSATBFED 
LJ  4  SATBFCD 

LJ  3   rCITVCR  SATBFCD  NOR  OSSATSFEO 
Lj  2  OeSATSFEO 
LJ  1    VBW06SATSFB} 

Lj    0  NOT/V>njCABU 

LJ  S  VERYSATBFED 

U  4  SATBFEO 

LJ  3  )CrT>CRS*TBFCONOR0ISaATBFED 

LJ  2  DBSATSFCD 

LJ  1    VERY  OSSATISFED 


Quality   of    Products    and    Servlcaa. 

16.  How  satisfiad  ara  you  VMth  th*  quality  of  NARD AC'S  sfltlaa£BJ2IBdUCia2 


0  NOT/^nxAai£ 

LJ  5  VERYSA-nsFED 

LJ  4   SATSFEO 

LJ  3    fCTTVER  SATBFED  NOn  DISSATSFED 

LJ  2  DBSAT15FCD 

LJ   1    VCRVOBSATOFED 


1 7.   Tha  quality  of  NAROAC'S  riata  nmcaagfui  swfvieaa. 


18.  The  controls  ovar  data  aaeurltv. 


LJ  0  NCfT>«nxAa£ 

LJ  5  VERYSATBFB) 
U  4  MTtSffiD 

LJ  3  ^c^>CRs*'^SFEDNonolss*'^sFCD 

LJ  2  OtSSATtSFED 
LJ  1    VERVOBSATOFED 

LJ  0  NCTAFnjcAaif 

LJ  S  VERYSATBFED 

LJ  4  SATWED 

LJ  3   NErTHERMTBFEONOROISSATtSFED 

LJ  2  nSSATBFED 

LJ  1    VERYOeSATBFED 
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AMflCMC  CUEHrSATKFACmN  SURVEY 


1 9.  Th«  reliabiliry  at  onlin*  and  batch  systams  (i.a..  th»  total  time  tti»  aysum  is  up). 


LJ    0  HJT  tfnjUBLE 

Lj   5   VEPYSATSFB) 

[J  4  SA-nSHED 

Lj   3    fCrrVCR  SATBFCO  NOR  0ISSAT6FE0 

LJ  2  ossA-nsFCD 

Lj  1    VERVOBSATBFED 


20.  The  accuracy  of  systam  documamatiorvmanuala. 


LJ    0  NOTAPPUCMLE 

Lj  5  VERYSAT1SFED 

LJ  4  SkTSFCD 

LJ  3  hCmCR  SATSFEO  NOn  OISSATBFCD 

LJ  2  DSSA-nsFCD 

[J  1    VERVC 


21 .  Th9  Dartormance  of  your  sysi«nt(s)  curramiy  running  on  NAROAC  compuiars. 


LJ  0  NOTAPnxAaLE 

LJ   5   V6RYSATBFB3 

LJ  4  SATBRED 

Lj  3  NEmCRSATBFCONORDISSATBFEO 

LJ  2  OBSATBFED 

LJ  1    VStYOaSATBFED 


22.  Tha  rasDonsanme  of  ontna  syslams. 


0  NcrrAPnjCAOf 

LJ  5  VERYSATSFE} 

Lj  4  SATSFED 

LJ  3  fCmCRSATBFCONOROISSATBFED 

LJ  2  OBSATBFCD 

LJ  1    VERYOSSA-nSFED 


23.  Whan  a  problani  assoaatad  with  a  NAROAC  product  or  aarvtca  is  rapoitad,  how  satiafiad 
ara  you  VMth  tha  Qualify  of  tha  romlutinn? 


LJ  0  Hori 

LJ  5  VERVSATISFB) 

LJ  4  SATBflED 

LJ  3   »CTr>CRMTBFE0NOR0(SSATSFCD 

LJ  2  OGSATWe} 

LJ  1   VERVOeSATBFED 


Tlmellneaa    of    Servleaa. 

24.  Han  taasfiad  ara  you  wtth  NARDACs  adharanca  to  oroiaet  aehadulaa  as  spacifiad  in  tha 
Suiamam  of  Work? 


0  NOTAPRXABLE 

Lj  S  VERVSATBFe> 

Lj  4  SATIsnB} 

LJ  3  rCrr>CRSATBFCDNORD*SSATBFED 

LJ  2  nsSATlSFED 

LJ  1   VERV06SATSFE) 
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mmAC  OJEW SATISFACTX3N  SURVEY 


25.  Adh«f  nca  to  production  pnocasana  schndiiHw 


0   NOT  APf>UCAaLE 

LJ  5   VEBYSATBFBJ 

LJ   4    SATISFED 

LJ   3    NEfTVeR  SATBFED  NOR  DISS*TTSFED 

LJ   2   OSSATBFe) 

LJ   1    VERYOBSATSFE) 


26.  Timeliness  of  nrt^ra*^  roonrta. 


0  NOTAPRXMLE 

LJ  5  VERVSATSFB) 

LJ  4  SATBFEO 

LJ  3   IcnVER  SATSFCD  NOR  OtSSATISFEO 

LJ  2  OBSA-nsro 

LJ  1    VERYOBSATQFe> 


27.  SotTware  MaintanancG  oaftermad  on  miaang  sysiams  (i.»..  maponanfWfiMS  in  corrwceng 
program  amrs). 


LJ     0   NOTA 

LJ    5   VERVSATBFH) 

LJ  4  SA-nsFCO 

LJ  3   rcmCR  SAHSFED  NOR  OtSSA-nsFCD 

U  2  DBSA-nSFCD 

LJ  1   VBtVOBSATSFED 


28.  Tlmalmass  of  notification  of  dalavs  in  outnuL 


0  NOTAPRXMLE 

LJ  5  VERVSA-nSFED 

U  4  SA-nSFCD 

LJ  3   ^CrTVCR  MTSFEO  NOR  OISSATBFED 

U  2  OBSAIvrCD 

LJ  1  vERYoesA-nsFE) 


29.  Timalinass  of  daliverabies. 


0  NOTAPRXABLE 

U  5  VERVSATBFED 

LJ  4  SATSFEO 

LJ  3  fCTTHER  SATISFED  NOR  OISSATISFCO 

LJ  2  OBSATBFCD 

LJ  1   VERVDBSATBFB) 


30.  Timalinaas  of  rasponsa  (or  a  raouagt  for  safvieaa. 


LJ  0  NOfTAPnjCAaf 

LJ  S  VERVSATSFB) 

LJ  4  MTBFED 

LJ  3  ICmCR  SATBFED  NOR  DISSATSFED 

LJ  2  OBSATBFEO 

LJ   1   VERVOSSATBFE) 
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lURCAC  CL£W SATISFACnON  SURVEY 

31 .  Wh«n  a  Drobl*m  assoaatad  wnh  a  NAROAC  product  or  sarvica  is  raponad,  how  saosfiad 
ara  you  >Mth  th«  timeliness  of  The  rasolmion? 


LJ     0   NOTAPPUCABLE 

LJ   S   VEBYSATBFtD 

[J  4    SATSFCD 

LJ   3    NEFTVEn  SATBFEO  NOR  OISSATSFED 

LJ  2   OeSATSFCD 

LJ  1    VERVOSSATBFED 


Comoeteney 

32.  How  satisfied  ara  you  with  tha  tachnicai  sufTs  level  of  aroartisa? 

(TeehnicMl  staff  inciubas  computer  spacuisB,  pmgn/nmtrs.  and  syst»m  anaJysts) 


U   0  Ncrri 

LJ  S  verysatbfed 
u  4  satbfeo 

Lj  3  fCrrvCR  SATBFED  NOR  OtSSATSFED 

lJ  2  OBSA-nsFED 

LJ   1    VERYDBSATBFED 


33.  Tha  technical  staffs  range  of  knowledge  in  ttiair  raaoaaiw  ADP  tiatda. 


lJ  0  NOTAfinxAaif 

U   5   VEnYSATBFS) 

Lj  4  SAVSfB) 

Lj  3   rcrTVCRS*TBFEDNOR(}<SSATSFE0 

LJ  2  DCBATBFED 

LJ  1   VERYOSSATBFEO 


34.  The  technical  suffs  knowledge  of  the  latest  develoomenta  in  thwr  raap«3iv«  ADP 
fields. 


35.  NAROAC's  ability  to  daaign  and  davalop  I 


Lsystama. 


LJ  0  tart 

Lj  S  vervsatsfb) 
lj  4  satsfcd 

LJ  3  KE(T>CRSATSFEDNOR0iSSA-nSFE0 

LJ  2  OBSATSnO 

LJ  1    VERYOBSATBFED 

LJ    0  NOTAPRXABIE 

LI  5  >CRrSATBFED 

LJ  4  SATBFCD 

LJ  3   rCmCR  SATBFEO  NOR  0ISSAT15FE0 

LJ  2  OBSATtSFED 

Ll  1   VBtVOSSATlSFED 


36.  Tha  NAROAC  partonnal  assignad  to  your  prejacL  (i.e..  do  th«y  damonstrata  tha 
Drtsoer  skill  level  to  partorm  tha  usk(s)  spadfiad  in  tha  Statamant  of  Woik. 


LJ    0  KOTAPFtJCAaE 

LJ  5   SCR/SATOFS) 

Lj  4  SA-nSFED 

LJ  3  »C(T>CR  SATBFEO  NOR  OBSATBFEO 

LJ  2  OBSATBFED 

LJ  1   VERYOBSATBFE} 
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I 


nAODAC  ciefrr  SAnsfAcrvN  SURVEY 


Coordination    and    Communlcatloni 


37.  How  saosfiad  ara  you  vMth  NARDAC's  undarstanding  of  tha  imoaa  on  your  command  for 

laie  or  inaccuraia  ofgflucis  or  servicfla? 


LJ   0   NOT  APPUCABLE 

[J  5  VERYSATBFB) 

Lj  4  SA-nSFCO 

LJ  3   ^CrTVCR  SATBFCO  NOR  OlSSATSnED 

LJ  2  DBSATBFED 

LJ  1    VERYOeSATBFED 


38.  NARDAC-.s  familiantv  with  vour  command's  misaon. 


LJ  0  NOfApnjcAaE 

LJ  5  VEnvSATBFED 

LJ  4  SATBRED 

LJ  3  NEtT>CR»TBFCDNOR0<SSATSFED 

LJ  2  OBSATBFEO 

LJ  1    VERVDSSATBFH) 


39.  Tha  leval  of  asajstanea  providad  In  praparing  oreooaals  fur  naw  profacB. 


0  NOTA^UCMLE 

LJ  5  VCnYSATOFED 

LJ  4  SA-rePED 

LJ  3   KETTVCRSA-nSFEDNOnOBSATBnED 

LJ  2  DSSATBFED 

LJ  1   VERVDBSATISFCD 


40.  Raxibilrtv  to  saOsfactorily  raspond  to  changas  in  your  spacification  raquiramants. 


0  NOfAPnxAaLE 

LJ   5   VERVSATBFB) 

LJ  4   SATBFCD 

LJ  3  rCmCRSATBFEONOROISSA'nSFEO 

LJ  2  06SATBFE0 

LJ  1   VERTOBSATBF^ 


41.  Courteous  and  orofassional  atHtuda  of  tha  NARDAC  oaraonnal  «wth  whom  vou  daaL 


LJ  0  Ncn-A(>nxAai£ 

LJ  5  VERYSATBFE} 

LJ  4  SATVPED 

LJ  3  NEfTVCR  SATVPEO  NOR  DISSA'nsFCD 

LJ  2  OBSATBFCD 

[j  1   VERVDSSATISFe) 


42.  Promulgation  of  genaral  ouroosa  information  about  what  is  happaning  around  NARDAC 
fi.e.,  now  harOwan,  sysiam  sotiwan,  application  sottwan,  training,  now  parsonnai,  me,) 
through  nawslattars,  alactrenic  buUatin  boards,  and  Cllani  Counbl  Maabngs. 


LJ    0  MTTAPfUCABLE 

LJ  5  VERVSATOFB) 

LJ  4  SATVCD 

U  3   rCmCRWnSFEDNOnOISSATSFED 

LJ  2  DeSATBFED 

LJ  1   VERY0B3ATS^ED 
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HARa*C  CLEW SATTSFACnOH  SURVFY 

43.   Th»  coortinaaon  and  (ollow-up  on  tasks/oroiects  assionad  lo  subcontractors. 


0   NQTAPfUCABLE 
[J   5   VERYSATBPED 
U  *   SATBFE) 

lJ  3  rcrrvcR  satbfed  nor  DissATisnED 

LJ  2    OBSA-nSFB) 

LJ   1     VEffrOeSATQFED 


44.  Th«  aUBty  of  NARDAC'S  t«chnK»l  «a«  to  eommunieala  in  elianl  tarms/lannuana. 
(T»chnic»l  staff  induOta  eompuiar  spaaatsa.  pmgramman.  and  systam  analysts) 


U    0  NOT4 

U  5  VERVSATBFE) 

LJ  4  SATBfO 

Lj  3   rCtTVCR  SATSFCD  NOR  OlSSATtSFEO 

[J  2  OSSATBFCD 

LJ   1    VERVOSSA-nSFCD 


Please  ptace  an  X  in  the  box  which  best  descnbes  the  relative  imponancs  of  each  of  the  following  oeriarmance  catena: 


1.     Accaaslblllty 


Lj  5  cwncM. 

LJ  4   MPOKllMr 
LJ  3  AVEMOE 

LJ  2  NOTft^otTrMr 

Li  1    RCLEVAMr 


TImailrwsa  tH 
Sarvie* 


LJ  5  cnmcM. 

LJ  4   MPORTAMT 
LJ  3  AVEMGE 

LJ  2  NcrTt«>OPrTAMr 

Lj  1    IWEL£VAMr 


2.     Syatam    Raaources 


LJ  5  cnmcM. 
LJ  4  t«>o(rrAMT 

LJ  3  AVEMOE 

LJ  2  NOTM>ORTM<r 

U  1    HBEV/MT 


6.     Competency 


LJ  5  cnmcAL 

LJ  4  ttPOmiMT 

LJ  3  AVEMGE 

LJ  2  NOT  •(•'OfTT'ANT 

LJ   1  fVCEVAMT 


3.     Coat    Management 


LJ  S  cnmcAL 

LJ  4   MPORTANT 
U  3  AVEMOE 

LJ  2  NOTaiPoeTM<r 
LJ  1  rmn-'rrr 


Coordination   and 
Communlcatlona 


U  5  CmiCAL 
LJ  4  MPOffTAMT 
LJ  3  AVEMGE 

LJ  2  NOTi«>om'AMr 

Lj   1    fWBEVAMT 


Quality   of   Produeta 
and    Sarvlcee 


Lj  S  CRmcAL 

LJ  4   MPORI-AMT 
Lj  3  AVEMGE 

Lj  2  NoraaxanTWT 

LJ  1    (VCLEVMr 


NtoT«_ 


.CDd»_ 


FIEASEPRNT 

Thank  you  for  completing  this  survey.  The  enclosed  envelope  has  been  provided  for  your  convenience. 
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APPENDIX   I.     PILOT  POPULATION 

Naval  Surface  Force,U.S.  Pacific  Fleet,  San  Diego 

Fleet  Accounting  and  Disbursing  Center  U.S.  Pacific  Fleet,  San  Diego 

Fleet  Hospital  Support  Office,  Alameda 

Naval  Station,Treasure  Island,  San  Francisco 

Naval  Air  Station,  Lemoore 

Naval  Air  Systems  Command,  Naval  Air  System  Command  HQ,  Washington,  D.C. 

Naval  Biosciences  Laboratory,  Oakland 

Naval  Strike  Warfare  Center,  Naval  Air  Station,  Fallon 

Shipbuilding, Conversion  and  Repair   USN,  Hunters  Point,  San  Franciscco 

Naval  Weapons  Station,  Concord,  CA 

Naval  Supply  Systems  Command,  Washington,  D.C. 

SIMA  NRMF  San  Francisco,  NAS  Alameda 

USS  Enterprise  CVN-65 

Naval  Base,  San  Francisco 

Navy  Public  Works  Center  San  Francisco  Bay 

Mare  Island  Naval  Shipyard,  Vallejo 

Naval  Education  and  Training  Financial  Information  Processing  Center,  NAS 

Pensacola 

Navy  Accounting  and  Finance  Center,  Washington,  D.C 

Naval  Air  Station,  Alameda 

Naval  Reserve  Force,  New  Orleans 

Naval  Air  Station,  Moffett  Field 

Service  Group  1,  Oakland 

Shipbuilding,  Conversion  and  Repair  USN,  Seattle 

U.S.  Maritime  Defense  Zone  Pacific,  Alameda 

Navy  Military  Personnel  Command,  Washington,  D.C. 

Naval  Aviation  Depot,  Naval  Air  Station,  Alameda 

Navy  Comptroller  Standard  System  Activity,  Pensacola 

Naval  Supply  Center,  Oakland 

Personnel  Support  Activity,  San  Francisco 

Military  Sealift  Command  Pacific,  Oakland 
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Naval  School  Physical  Distribution  Management,  Oakland 

Naval  Medical  Command,  Northwest  Region,  Oakland 

Special  Boat  Unit  Eleven,  San  Francisco 

Naval  Postgraduate  School,  Monterey 

Naval  Facilities  Engineering  Command,  Alexandria,  VA 

USS  Kiska 

Naval  Supply  Center  Puget  Sound,  Bremerton,  WA 

Joint  Military  Postal  Activity  Pacific,  San  Francisco 

Marine  Barracks,  Naval  Air  Station,  Alameda 

Sacramento  Army  Depot,  Sacramento 

Navy  Astronautics  Group,  Point  Mugu,  CA 

Chief  Preservation  Assistance  Branch,  San  Francisco 

Navy  Resale  and  Services  Support  Office,  Field  Support  Office,  Oakland 

Navy  Legal  Service  Office,  San  Francisco 

Mare  Island  Naval  Shipyard,  Vallejo 

Naval  Station  Mare  Island,  Vallejo 

Tactical  Electronic  Warfare  Squadron  One  Twenty  Nine,  NAS  Whidbey  Island 

Defense  Subsistence  Region  Pacific,  Alameda 

Naval  Air  Station,  Fallon 

Naval  and  Marine  Corps  Reserve  Center,  San  Francisco 

Naval  Electronic  Systems  Engineering  Center  Vallejo 

NARDAC  Newport 


97 


APPENDIX   J.     PILOT  QUESTIONNAIRE 


MPOAC  CUEtfT  iATafACTIOM  SIMVWT 

PLEASE  PLACE  AN  X  IN  THE  BOX  WHICH  BEST  DESCRIBES  YOUR   LP/et   OF  SATISFACTION     WITH  EACH  OF  THE 
FOLLOWING  ITEMS.  {AddUionai  comments  may  be  made  on  ihe  last  page). 

USE  THC  FOLLOWING  SCALE: 

H/A   NOT  APPUCABLE      5  EXTBaCLY  SATTSFCO  4  SATISFIED       3   NIUTTttL       2  DISSAT1SFI60        1    EXTPEMaY  OBSATISF60 

Accessibility. 

(£as9  of  obtaining  a»rne») 

1.  Tha  availabilitv  o<  vour  NARDAC  ooint  al  contact.  N//0        sQ     aQ     oO     2O     iQ 

2.  Tha  nmoidurB!!  rnmjimd  inNAVDACtNST  5230. 1  tor  ofttainino  vanom  NARDAC  OfeduOi         N/aD         sD     -lO     oD     20      lU 
and  sarvicaa. 

3.  TTia  currant  houni  nf  noaration  at  NARDAC.  N/aO  sQ     4O      ^Q     2D      iD 

4.  Tha  ohvaicai  location  ai  NARDAC  rataliva  ta  vouf  command.  N/aO        9_J     4O     3LJ     2O     lU 

5.  Tha  auantrtv  at  vigrtor  oaf»in(;i  soaeaa  at  NARDAC.  N/aQ         sD     4D     oD     20      iQ 

Svstam    RasQureaa. 

(£ncompaaa»a  natOwan,  sottwan.  and  talmprocaisftg  taetinology) 

S.      TTia  cuiram  eomoutaf  hantwara  caoabrlitlea  at  NARDAC.  N/aO        SLJ     4O     XJ     2O     iLJ 

(i.»..  maintrain*,  mmKomputan.  mKrocDmputan) 

7.       Tha  ahilitv  frf  NARDAC  ta  mmM  vour  rvninrnirf  autrwrtatad  Ififormation  watam  raouiremanW-      N/M_J  sLJ     ^LJ      3Lj     2i_i      lLJ 

a.       Tha  curranev  and  varralv  of  aoWwara  oaekaqaa  avilabla  at  NARDAC.  N/aO         sO     ^LJ     3LJ     2O     lU 

9.  Tha  abilitv  of  NARDAC  to  omvida  adaouata  telaoroeasainq  sarvicaa  to  mat  vouf  N/aO        SLJ     4O     3O     2O     lLj 
command's  oparationai  na«ds. 

Cost    Manaoamant. 

(Includaa  controls  ovar  ptojaex  budgata,  rata*  chargad  lor  sarvicaa,  biBng  proeadtiraa, 

and  pnca  stnicturaa) 

10.  NAROAC-3  ability  to  pravida  an  accurata  assMsmant  of  both  N/aO        sO    4O     aU    2D     iQ 
tha  costs  and  tha  schadula  for  a  projact 

11.  NARDACs  adharanea  to  omiact  budoata.  N/aO         sO     ^O     sD     2O     iD 
1^     Tha  abilitv  of  NARDAC  to  adaouatahr  a«plain  prnjaet  eoata.                                                         N/aO        sO     ^O     gO     2O     iD 

13.  Ttya\ansAf*^'*f^^G*Ot\aiq^Omd».Rapon(La.,aaartoramdandundantand).  N/aO        sO    4U     gO    2O     iD 

14.  ThaacCUOQiaf  NAROACsChargabackRapofL  N/aO        sO    iH     gO     2D     iD 
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I         HAROtC  CUEMT  SAWAC-nOM  SUItVWY 
USE  THE  FOUOWINO  9CAL£: 

N/A   NOT  APPtlCABU        5   EXTWEHCLY  SATTSPED  4   SATKFCD        3   MEUTlUt       2  DISSATISRED         1    EXTBEI^LY  OBSATISFEJ 

15.  Th«  cuasnuaiaa  NARDAC  chargas  tor  products  and  sarvioM.  N/aO         sCJ     4LJ     gO     2D     lD 

16.  flfi2ituM  in  charging  for  NARDAC  products  and  sarvicM.  N/aO        sO     4O     sO     2D     lD 
(^•.3.,  tix»d-pnc»  contract,  mrtd  r»t»  structun,  0tc). 

Quality   of    Products   and   StvIcb. 

(NARDAC's  ability  to  daiivor  rMabla  s«rvc»s  and  axcaiiant  products) 

17.  Tha  Quality  of  NARDAC  devetoped  application  system  products.  N/aD        sD     4D     sD     2D     iD 

18.  The  Quality  of  NARDACs  data  arocessina  sarvios.  Hltd         sD     ^D     3D     2D     lD 

19.  The  secuntv  controls  oyer  vour  data  tfiai  is  maintained  on  NARDAC  eomnutara.  ^4/AO         sD     4O     3LJ     2D      lD 
(i.a.,  physical  sacunty,  usar  id/passwords,  ate.) 

20.  The  raliabilrty  of  NAROAC  onlma  and  batctt  systems  (i.a..  tha  total  tima  tha  aystam  is  up).  N/aD  sD  4D  3U  2D  lD 

21.  The  accucac^  of  NAROACs  system  documantatiort/manuaii.  h4/AO  sO  4D  3O  2D  lD 

22.  Tha  iflsa&aaajims  of  NARDAC's  online  systems.  N/aO  sO  4D  3D  2D  lD 

23.  The  quality  of  tha  rasolulion  to  «  orobtam  Maoeaatad  with  a  NARDAC  ntndua  nr  umivirm  N/aD  sD  4D  3D  2D  lD 

Timeliness    of    Sarvleaa. 

(NARDAC's  ability  to  ba  punctual  and  '  schadula  consaantious'  with  its  products  and  sanncaa) 

24.  NARDACs  adherence  to  project  schedules  aa  anaeifiad  in  ttia  N/aD        sD     4D     3D     2D     iD 
Protect  Request. 

25.  The  ability  of  NARDAC  to  adhere  to  production  processing  schedules.  N/aD        sD     4D     3D     2D     lD 

26.  The  frequency  of  NARDAC  progress  reooits.  N/aQ        sD     4D     3D     2D     iD 

27.  The  timely  notifieation  of  dalava  in  ompul  N/aD        sD     4D     3D     2D     iD 

28.  Timalinaaa  of  deliyarablaa  from  NARDAC  N/aD         sO     4D     3D     2D      iD 

Responslveneae 

(How  quckly  NARDAC  raads  or  rasponds  to  dant  naads) 

29.  Software  maintenanea  narformad  on  existing  NARDAC  synems  (La.,  raaponsivanaas  in  N/aLJ        SLJ     4D     £2     2D     lD 
correcting  program  arrors). 
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I      NAROAc  cuesT  SA-nsf  ACTION  suRver 

USE  THE  TOLLOWINQ  SCALE: 

hVA  NOT  Appuc*ai£     5  Exnmm.y  satbred       4  satbhed     3  neutwu.     2  oissATisFeo      i  extoejuB-y  obsavsped 

30.  Thn  raspnn«iv«n««ii  of  NARDAC  la  a  faouBSl  for  snrvirgt  N/aLJ  5LJ     4LJ     sO     zLJ      iU 

31 .  Th«  raponsfvanAss  of  NARDAC  to  raaolving  a  raponad  problam  associalad  with  a  product        ti/fLJ        SLJ     aLJ     3LJ     2LJ     lU 
or  sarvica. 

32.  Tha  flaifibilrlv  of  NARDAC  to  ehanoaa  in  vour  nmiacl  raouiramanta.  N/aLJ         5lJ     4LJ     3LJ     2LJ      iLJ 

33.  The  Iflvpl  nf  a«i«;i«!tanga  pmvidad  in  pfnoarinq  nmnosals  for  nam  PfOfaeta.  N/aLJ         sLJ     4LJ     3LJ     2l_l      lLJ 

Staffing. 

(Encompassaa  tfia  r»aourc»tultfss  and  r»quisit0  quaificaions  of  NARDAC  p»rsonn»l) 

34.  Tha  Qvarall  ranoa  of  exoartisa  twthin  tha  NARDAC  ofoanaalion.  N/aO        SlJ     4O     3L_J     2LJ     lU 

35.  Tha  lavel  of  aroartlsa  of  tha  NARDAC  naraonnal  «»i»h  whom  vou  d—L  N/aO         sO     4O     sO     zD     iD 

36.  Tha  abilitv  of  NARDAC  oarionnal  to  kaao  abraaat  of  the  latast  developmems  in  thair  N/aQ        sQ     4O     30     2O     lU 
raspactiva  functional  oraa. 

37.  Tha  eouftaoua  and  orofaasional  attituda  ot  tha  NARDAC  oafaonnal  with  whom  vou  daaL  N/aO        SLJ     4O     sO     2LJ     lU 

38.  Tha  tachnical  staff's  ability  to  dasign  and  davaiop  ereative  and  Innovativa  svatama.  N/Ai_l        sLJ     4LJ     sLJ     2LJ     lLJ 
(Technical  statf  mctudss  oomputtr  apmcmlists.  programmis.  and  systam  analysts) 

39.  Tha  tachnical  staff's  ability  to  satsfactority  analvre  and  document  vour  omiael  N/aQ        sO     4LJ     a_J     2O     iLJ 
reouirements. 

40.  Tha  abilitv  of  NARDACs  technical  staff  to  communicate  in  diam  tarnis/lanouaoa.  N/aD        sD     40     sD     2D     iD 

General    Bustnaas    Practlcaa. 

(The  manner  in  which  NARDAC  conducts  buaineas  with  Us  dents) 

41.  NARDACs  familiantv  with  vour  command's  mission  aa  it  raijtaa  to  data  ofocaaaina.  N/aO        sO     4Q     3LJ     2lJ     iU 

jD         sD     40     sD     20      iD 

43.     Tha  coordination  and  follow-up  on  taak«/nmi«ets  assigned  to  gubetintraaors.  N/aLJ         sLJ     4LJ     3LJ     2LJ     lLJ 


42.     NAHDACs  understanding  of  the  imoact  on  vour  command  for  late  or  inaectjrate  oroductt  Wi 

or  sarvicas. 


44.     Promulgation  of  ganaral  oumoaa  information  about  what  b  happening  around  NARDAC  N/aLJ         SLJ     4LJ     sLJ     2LJ     iLJ 

(i.e..  new  hardware,  system  software,  applcaton  software,  training,  new  personnel,  etc) 
through  naMslattars,  electronic  bullatin  boards,  and  Client  Couml  maatirtgs. 
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4     NAROAC  CUCNT  SATtafACTlOM  SURVTT 


PLEASE  PLACE  AN  X  IN  THE  BOX  WHICH  BEST  DESCRIBES  THE  ReiATIve    IMPORTANCE  TO    YOUR 


ORGANIZATION   OF   EACH   OF  THE   FOLLOWING    PFRFORMANCE    CRITERIA    AS   IT  RELATES  TO  CUENT 
SATISFACTION. 

USE  THE  FOLLOWING  SCAL£: 

5  CRITICAL 

4  IMPORTANT 

3  AVERAGE         2  NOT  miPORTAMT 

1  IRRELEVANT 

1.      Accaaslblllty 

(Eas0  ol  obtwnmg  s»rvie») 

sD 

40 

3O 

n 

u 

2.      System    Raaourc** 

(Encomoassts  harxJwara.  softwvo,  and 
t»l»proc9ssing   technology) 

sD 

40 

3O 

n 

iD 

3.      Cost    Managatnsnt 

(Includos  controls  ovar  projact  budgats,  ratas 
chaigad  tor  sanncas.  billing  procaduras.  and  pnca 
struauras) 

sD 

40 

3O 

20 

iD 

4.      QualKy   of   Products  and   Sarvlcaa 

INARDACs  ability  to  dalivar  ralmbla  sanncas  and 
axcallant  products) 

sD 

4O 

30 

20 

iD 

5.      Timallnaaa   of   Sarvleas 

(NARDAC's  ability  to  ba  punctual  and  'sct)aduia 
conaciantjous'  with  its  products  and  sar^ioas) 

sD 

4O 

3O 

20 

iD 

6.      Raaponalvsnass 

(How  quickly  NARDAC  raaas  or  rasponda  to 
cmni  naads) 

sD 

4O 

3O 

20 

iD 

7.      Staffing 

(Encompassas  tha  rasourcalulnass  and  raquisHa 
qualifications  of  NARDAC  parsonnal) 

sD 

40 

3O 

20 

iD 

6.      Gsnaral    Buslnsss    Praetlcss 

(Tha  mannar  in  ¥ituch  NARDAC  aonducia  busnaaa 
with  Its  cliants) 

sD 

4O 

3O 

20 

iD 

Commanta: 

l^ma 

Qada 

Tatetana 

REMEPHNT 

____N«ma  of  your  NAflDAC  POC 

Thank  you  for  completing  this  survey.  The  enclosed  envelope  has  been  provided  for  your  convenience. 
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APPENDIX    K.  REVISED  QUESTIONNAIRE 


fai>D*c  custn  sAnafAcnoM  survwy 

PLEASE  PLACE  AN  X  IN  THE  BOX  WHICH   BEST  DESCBIBES  YOUR  LEVEL   OF  SATISFACTION     WITH  EACH  OF  THE 
FOLLOWING  ITEMS.  (Additional  comments  may  be  made  on  the  last  page). 

USE  Txi  POLLOW1NO  scale: 

N/A  NOT  APnjCA&E  5  eaveHELf  SATISFCO  4  SATISFED  3    NEUTPAI.         2  OtSSA-nSFEO  1   eCTREhCLY  0ISSAT1SFEO 

Accasalbllltv. 

(Easa  ot  otttaming  S0fvie») 

1 .      Th*  availability  of  your  NAROAC  pomt  o(  contact.  N/< 


'/jn   n  n  n  n  u 


2.  Tha  PTQCodufaa  raouirad  JnNAVDACINST  5220.1  for  obtaining  vahoua  NARDAC  oroducta         N/aQ         sD     -O     gD     sD      iQ 
and  sarvicas. 

3.  Tha  currant  houra  of  ooaration  at  NARDAC.  N/aQ         sQ     4Q     3Q     ^Q     iD 

4.  Tha  Physical  location  of  NARDAC  ralaliva  to  vour  command.  N/aO         sLJ     ^LJ     gLJ     2LJ     iLJ 

5.  Tha  Quantity  of  visitor  oafkino  soacaa  at  NARDAC.  N/aO         sO     «LA     3_]     2Lj      iLJ 

Svatam    Raaoureaa. 

(Encompassas  haittwarm,  sottwmn,  and  taiaptoctssgtg  tachnohgy) 

5.       Tha  currant  comoutar  harrtYtara  panahilitin^  at  NAROAC.  N/aQ        sO     4LJ     3LJ     2LJ     lLJ 

(i.:.  mainfmn*.  mvuoomputarf,  miaxicomputan) 

7.       Tha  ability  of  NARDAC  In  maat  your  ofoiectad  automatad  Infofmation  systam  feouifamanta.      N/A|_J         sLJ     4LJ     3LJ     tLJ     ll—l 

3.       Tha  currency  and  vanntv  of  softwara  oadtanaa  availabla  a«  NAROAC.  N/aO         sO     ^LJ     3O     2O      iLJ 

9.  Tha  ability  of  NAROAC  to  oroyida  adaouata  talaorocassino  sarvcaa  to  maat  vour  N/aO        SlJ     "LJ     3L_1     2LJ     iLJ 
command's  oparaiionai  naads. 

Cost    Manaoamant. 

(Incluaas  controls  ovarpmjaei  budgats,  ratma  cha/gad  farsarvioaa,  biUing  pmcaduraa, 

and  pnca  stmauraa) 

10.  NAROACs  ability  to  proyida  an  accurata  asaMamam  of  both  N/aO        sO     4O     3O     2O     lU 
tha  costs  and  tha  schadula  tor  a  proiact 

11.  NARDAC-s  adhawnea  IB  nmiaet  hurinam.  N/aO  sO     40     gO     2O      iD 

12.  Tha  abilJN  et  NAROAC  to  adanuatalw  axnlain  nmiart  m«m  N/aQ         sQ     4Q     gO      2D      iU 

13.  '{haismalotf*M^'^»Ci\»ig'b*ckHaqon(La..aatrtoimdandundaf$t»tdf.  N/aO        sQ    4Q    3O    2O     lD 

14.  Tha^suLacxofNAROACsChargabackRaport.  N/aO        sO     40     gO     20     iD 
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NAROAC  CUBHT  SATlSfACnOM  SUHVtr 
USE  THE  FOLLOWINO  SCALE: 

N/A  NOT  APPUCABlf  5   EXmEVELY  SATEFED  4  SATISFED  3    fCUTRAL         2  OSSATISFED  1   EXTDEkCLY  OSSATSFED 

15.  Tha  cunanuataa  NAROAC  Charges  tor  products  and  MrvcM.  N/aO        sO     4LJ     3LJ     aJ     lD 

16.  ElasMJtX  in  charging  for  NARDAC  products  and  sarvicM.  h4/AO        sO     4U     3U     2G     lU 
(:g.,  fix0d-pnc»  contrma.  t»r*d  rate  struetun.  «ie.). 

Quality    of    Produela   and    Srvleaa. 

(NARDAC's  ability  to  d»liv»r  nUabia  sarvKas  and  axcatlant  ptoducts) 

17.  Tha  Quality  of  NARDAC  davalooad  aoplicaiion  wsiam  ofoducls.  N/aD         sO     4U     sG     2D      iU 

18.  Tha  Quality  of  NARDACs  sarvieas  N/aO        sO     iD     3O     2D     iD 
(Sarvioas  induda  traming.lntormation  Rasourca  Cantar  (IRC)  assistanca, 

procassing.Cliant  Uaison  sarvicas.studias.atc.) 

19.  The  sacuritv  controls  ovar  vour  data  thai  la  maintainad  on  NARDAC  comoulara.  N/aO         sO     4LJ     3LJ     2—1      lLJ 
(i.a.,  physcal  sacunty,  usar  id/pasawords,  ate) 

20.  Tha  raliabilitv  of  NARDAC  online  and  batch  avatama  Ha.,  tha  total  tima  tha  avstam  a  uol.  N/aO  sQ  ^Q  sD  2D  lD 

21.  The  accuracy  of  NARDACs  syatam  doajmentation/fnanuaia.  N/aO  sO  4D  £J  £J  iD 

22.  The  resDonae  lime  of  NARDACs  online  ayatema.  N/aD  sD  4D  3D  2D  iD 

23.  The  Quality  of  tha  resolution  to  a  problem  assoaatad  with  a  NARDAC  product  or  servioe.  N/aO  sO  4LJ  3O  2D  lU 

Timeliness    of    Servtcea. 

(NARDAC's  ability  to  ba  punctual  and  '  schadula  oonadantious'  with  ks  ptoducts  and  sarvtcaa) 

24.  NARDAC's  adherenea  to  omiact  sehadulaa «« aoecified  in  the  N/aD         sD     4D     sD     2D     iD 
Profect  Request. 

25.  The  abiiiW  of  NARDAC  to  adhem  to  omduetion  nmcwsino  «^edulaa  N/aD         sD     4D     sD     2D     iD 

26.  The  frequency  of  NARDAC  Ofooress  reports.  N/aD        sD     4D     sD     2D     iD 

27.  The  timely  notifiealion  of  delava  in  outntiL  N/aD        sD     4D     3D     zD     lD 

2B.    Timeliness  of  servieea  from  NARDAC.  N/aD        sD     4D     sD     2D     iD 

(Services  include  trajning.lnformatbn  Resource  Center  (IRC)  assistance, 

processing, Client  Liaison  servicas.studies.etc.) 
Resoonslveneaa 
(How  quKkly  NARDAC  raacts  or  raaponds  to  daitt  naada) 

29.  Software  maintanance  performed  on  existing  NAROAC  systems  ^iLe.,  responsfl>«fiess  in  N/aO        SLJ     4l_l     sLJ     2lJ     iLJ 
oorraamg  program  arrors). 

30.  The  responsiyeness  of  NAROAC  to  a  reouast  tor  saryices.  N/aO        SLJ     4LJ     3D     2D     lU 
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MAROAC  CU£MT  SATISFACTIOM  SUHVtY 
USE  THE  FOLLOWINO  SCALE: 

NTA  NOT  APPUCASLE  5   EXTTEkCLY  SA-nSFCO  4  SATISFED  3    NEl/TRM.         2  OSSATISFED  1   EXmEtCLY  aSSATSFED 

31.  Tha  reponsivanass  o(  NAROAC  to  rMOlving  a  rapcrtad  problam  assoctotsd  with  a  product         N/aLJ         sO     ^.J     gLJ     ZlJ     1lJ 
or  sarvica. 

32.  Tha  UssJbilJt^  of  NAROAC  to  ctiangas  in  your  praiact  raqwramanti.  N/aO         sO     4l_I     3LJ     2O     lU 

33.  Tha  iBvel  of  assistance  ptovidad  in  prapanng  proposals  tor  naw  projacts.  N/a1_]         SLJ     4_1     3LJ     sLJ     iLJ 

Staffin?. 

(Encompasses  th»  nsoufcafulrmas  and  nquisM»  qualSications  of  NARDAC  pmnonnal) 

34.  Tbaovarall  ramya  of  aitfMiniSB  witt<in  ttM  NABDAC  oroangalion.  N/aLJ         sO     4LJ     3LJ     2LJ     IlJ 

35.  The  laval  of  axpartisa  of  ttia  NARDAC  oaraonnal  witti  ^^vom  vou  daaL  N/aO         sO     4U     3U     2U     iD 

36.  Tha  abilitv  of  NARDAC  ofaonnal  to  kaao  atifaaat  of  the  latest  davatoomams  in  thair  I^aO         SLJ     4LJ     3LJ     2lJ     iU 
raspactiva  tunctjonai  araa. 

37.  Tha  eourteous  and  prof  easionaJ  attitude  dt  tha  NARDAC  oafaonnat  w«h  whom  vou  d— I.  N/aO         sO     4LJ     SLJ     2O     lLJ 

38.  Tha  tachnicai  staff's  ability  to  design  and  develop  craativa  and  innovative  svatema.  N/aLJ         sLJ     4LJ     3LJ     sLJ     lLJ 
(Tacftnca/  statf  mdudaa  computer  speaaiists.  pmgnmmts,  ana  systam  analysts) 

39.  The  tachnicai  staff's  ability  to  sattsfactonfy  analyze  and  document  vour  oroiect  N/aLJ        SLJ     4LJ    ^LJ     zLJ     iLJ 
reouiremanls. 

40.  Tha  abilih^  of  NARDACs  tachnicai  ataff  to  eammunieate  in  dient  terms/lanouaoe.  N/aLJ         sO     4LJ     3lJ     2LJ     lLj 

General    Buslnaaa    Praeticea. 

(The  manntf  in  which  NARDAC  conducts  businaaa  with  Us  diants) 

41.  NARDACs  familiantY  with  your  command's  mission  aa  it  ralaiaa  to  data  pmca«ain9.  N/aO        sO     4O     3lJ     a_I     lU 

'aD   n  u  n  n  u 

43.     The  coordination  and  follow-up  on  tasjca/ramiacls  assigned  to  subenntraptnrc  N/aLJ         Si— I     4LJ     sLJ     2-1     1I— I 


42.     NARDAC'g  underaandinn  of  tha  imoaa  on  vour  eommand  for  lata  or  inaccurate  nmduaa  m 

or  services. 


44.    Promukiation  of  oanaral  ouroosa  information  about  what  is  haooanma  around  NARDAC  N/aO        sLJ     4LJ     a_l     zLJ     l[_l 

(i.a.,  naw  hantwara,  sysiam  sottwara.  appiicatan  sottwara.  training,  naw  paraonnal,  ale.) 
through  newsiattars,  elactronc  bulletin  (wards,  and  Client  Counal  meetings. 
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MAHDAC  cuorr  sATKrAcnoH  aimvrr 


PLEASE  PLACE  AN  X  IN  THE  BOX  WHICH  BEST  DESCRIBES  THE  RELATIVE    IMPORTANCE  TO    YOUR    ORGANIZATION 
OF  EACH  OF  THE  FOLLOWING  PERFORMANCE    CRITERIA    AS  IT  RELATES  TO   CUENT  SATISFACTION. 


USE  THE  FOLLOwma  scale: 

1.       Aceassiblllty 

(Baso  of  obtaining  sarviot) 


Syatam    Rasourcaa 

(Encompasses  hardwara,  soitwan.  and 
taiaprocassing  tachnoiogy) 

Cost    Managemant 

(Includes  controls  ovar  pio/act  txxigats.  ratas 
chargad  tor  san/icas,  billing  procaouras,  and  pnoa 
struauras) 

Quality  of   Products  and   Sarvlcaa 

INARDAC's  ability  to  dalivar  ralmUa  sarvioas  and 
axcallant  products) 

Tlmallnaas   of   Ssrviess 

(NARDACs  ability  to  ba  punctual  and  'schaduta 

oonsaantious'  with  its  products  and  satvcas) 

Raaponaivanass 

(How  quickly  NARDAC  raacts  or  rasponds  to 
aant  naads) 

Staffing 

(Encompaasas  tha  raaourcafulnass  and  raquiaita 
quaiHications  of  NARDAC  parsonnaO 

Gansral    Buainass    Practlcsa 

(Ttia  mannar  in  wtich  NARDAC  conducts  busmat* 
with  Its  dianta) 


ScnrriCAi. 
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40 
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3  AVERAGE 
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Commantsi. 


f>to™_ 


_CodB_ 


FIEASEPRNT 


Ngvna  cf  yojr  organzaian. 


.Nwna  of  ya#  NARDAC  POC_ 


Thank  you  for  completing  this  survey.  The  enclosed  envelope  has  been  provided  for  your  convenience. 
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